PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation

Krishna Rao; Andrew Dumit; Shaena Ulissi; Jacob Feintzeig; P. James Joyce; Daniel Frank; Steven Watson; Jonathan Glidden

cs.AIAug 27, 2026

PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation

Authors: Krishna Rao, Andrew Dumit, Shaena Ulissi, Jacob Feintzeig, P. James Joyce, Daniel Frank, Steven Watson, Jonathan Glidden,

Organizations: Watershed Technology, Inc.

Abstract

AI systems are being deployed on high-stakes, domain-specific workflows that demand correctness not just in the final output, but at every intermediate step. One such workflow is estimating a product carbon footprint (PCF), the greenhouse-gas emissions attributable to a physical product. AI agents are increasingly being used to generate PCFs, but existing evaluations score either total emissions (hiding error sources and cancelling mistakes) or sub-tasks in isolation (missing compositional interactions). We introduce PCFBench, the first benchmark to carve PCF modeling into independently-evaluable tasks that require decomposition, retrieval, ontology matching, and numerical extraction. It comprises 614 expert-labelled items across six tasks. Together they probe reasoning under under-specification, conflicting context, and numerical constraints. Across eight frontier LLMs from four providers, no single model dominates. Although the strongest models estimate total product emissions within 2 times of declared totals on 77% of products, this rate drops to 37-58% when the PCF is generated step by step, with only 45-75% obeying mass conservation. These failures undermine the transparency practitioners need to compare products and drive decarbonization. We release the dataset and evaluation harness to support targeted progress.

Figures & tables

Figure 1: Decomposed PCF creation. Orange brackets denote tasks covered by PCFBench ; grey is deterministic.

	Task 1	Task 2	Task 3	Task 4	Task 5	Task 7
	Decompose	Triage	Map	Material extraction	Energy extraction	Total kgCO 2 e
Babbitt et al. 2020 [ Babbitt et al., 2020 ]	✓	✗	✗	✗	✗	✗
Parakeet [ Balaji et al., 2025 ]	✗	✗	✓	✗	✗	✗
Sustain-LLaMA [ Kumar et al., 2025 ]	✗	✗	✗	✓	✓	✗
Carbon Catalogue [ Meinrenken et al., 2022 ]	✗	✗	✗	✗	✗	✓
Eco-Amazon [ Spillo et al., 2026 ]	✗	✗	✗	✗	✗	✓

Table 1: Data coverage of PCFBench versus prior datasets. ✓ = task-relevant ground-truth is publicly available; ✗ = no.

Task	Items	Source	Input	Output	Metrics
1. Decomposition	94 products	EPD	Product description	Bill of materials	$F_{1}$ , precision, recall, Kendall $\tau$
2. Triage	200 items	Hand-generated	Candidate + root material context	Map / decompose	Accuracy, $F_{1}$
3. Mapping	109 materials	Hand-generated	Material name ( $\pm$ context)	ecoinvent activity	Exact-match, relevant-substring, banned-substring violation
4. Material rates	22 questions, 55 claims	PDF	Question + source PDF	kg + evidence	Claim $F_{1}$ , supported $F_{1}$ , P90 $\|$ RE $\|$ , hallucination
5. Energy rates	14 questions, 34 claims	PDF	Question + source PDF	kWh/MJ + evidence	Claim $F_{1}$ , supported $F_{1}$ , P90 $\|$ RE $\|$ , hallucination
7. Validation	175 products	EPD	Product ( $\pm$ comp., region)	kgCO 2 e	Median $\|$ RE $\|$ , within-2 $\times$ , within-5 $\times$

Table 2: PCFBench dataset overview and per-task evaluation metrics. Task 6 (multiply and sum) is deterministic and not evaluated. Exact metric definitions in Appendix F .

Figure 2: PCFBench dataset overview. (a) Category coverage : product categories across the six evaluable tasks. Number and shading denote claim/item counts; colour encodes $\log(\text{count}+1)$ . Furniture, Construction, Vehicles, Electricity-and-fuels, and Services are bucketed into “Other” (Appendix C ). (b) kgCO 2 e distribution : log-scale histogram of kgCO 2 e per kg across the 175 EPDs in Task 7.

Model	Task 1	Task 2	Task 3	Task 4	Task 5	Task 7	Task 7
	Decompose	Triage	Map	Materials	Energy	Direct	Compositional
	$F_{1}$	Acc.	EM	$F_{1}$	$F_{1}$	$\leq 2\times$	$\leq 2\times$
GPT-5.5	0.73 $\pm$ 0.02	0.68 $\pm$ 0.03	0.80 $\pm$ 0.04	0.31 $\pm$ 0.07	0.29 $\pm$ 0.06	0.77 $\pm$ 0.03	0.55 $\pm$ 0.04
GPT-5.4-mini	0.70 $\pm$ 0.03	0.62 $\pm$ 0.03	0.75 $\pm$ 0.04	0.29 $\pm$ 0.08	0.48 $\pm$ 0.10	0.60 $\pm$ 0.04	0.50 $\pm$ 0.04
Claude Opus 4.6	0.78 $\pm$ 0.02	0.66 $\pm$ 0.03	0.85 $\pm$ 0.03	0.27 $\pm$ 0.09	0.29 $\pm$ 0.05	0.73 $\pm$ 0.03	0.58 $\pm$ 0.04
Claude Sonnet 4.6	0.74 $\pm$ 0.02	0.72 $\pm$ 0.03	0.82 $\pm$ 0.04	0.39 $\pm$ 0.08	0.39 $\pm$ 0.06	0.63 $\pm$ 0.04	0.50 $\pm$ 0.04

Table 3: PCFBench headline results. Task 1

F_{1}

, Task 2 accuracy, Task 3 exact-match (with-context detail in Appendix F.4 ), Task 4–5 claim-

F_{1}

, Task 7 within-

2\times

of declared kgCO 2 e; exact metric definitions in Appendix F . Tasks 2 and 3 report scores from the agentic harness (single-shot variants in Appendix H.5 ). Direct Task 7: directly predicts the kgCO 2 e of the EPD. Compositional Task 7: an agent assembles kgCO 2 e compositionally by chaining PCFBench ’s task agents on the same EPDs (pipeline spec in Appendix F.8 ). Both modes receive the same input: the product’s name and description.

\pm

values are bootstrap standard errors from 1,000 item-level resamples (protocol in Appendix F.9 ). Parakeet ‡ is a specialized baseline targeting mapping only.

Figure 3: Baseline model performance on PCFBench . (a) Task 7 kgCO 2 e within-

2\times

rate: direct (one-call) vs compositional (Tasks 1–5 chained, summed via deterministic ecoinvent EFs). Both modes start from product name and description; the compositional pipeline derives composition through Task 1. (b) Per-model relative-error distribution; upper half-violin = direct, lower = compositional. (c) Mass-conservation violation rate (compositional only). (d) Task 2 triage and Task 3 mapping: non-agentic vs agentic accuracy. (e) Accuracy vs increasing context across Tasks 2, 3, 7. (f) Accuracy vs. cost.

y

-axis is the mean of each model’s within-task min–max-normalised headline scores: for each of the six tasks (T1, T2, T3, T4, T5, T7) we rescale the eight models’ headline metric to

[0,1]

via

(v-v_{\min})/(v_{\max}-v_{\min})

across the eight models, then average a model’s six rescaled scores. Error bars are the across-task standard error of that mean.

x

-axis is mean USD spend per benchmark item.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Task 1 BOM size distribution across the 94 products.

Figure 5: Tasks 4–5 dataset characterisation. Left: OCR’d source-document length per item, in thousands of characters (log scale). Median 30.5k, P90 98.1k chars. Centre: number of verbatim evidence quotes per ground-truth claim (mean 2.6, max 8, 227 total). Right: ground-truth value scale per unit family (top-8 by claim count, log scale). Material claims (%, mass%, kg/kg, g/kg) cluster within

\sim

two orders of magnitude; energy claims (MJ/kg, kWh/kg) span

\sim

three.

Figure 6: Tasks 4 and 5 unit-family distribution across the 55 material and 34 energy ground-truth claims. Cross-family energy units canonicalise to MJ/kg via documented conversion factors at score time.

Figure 7: Annotation example for Tasks 4–5. The full source page is page 6 of Primary Industries and Regions South Australia (2017) [PIRSA] , a one-page Farm Facts calculator that projects an annual liming rate forward to a ten-year amortised application. Both candidate claims are grounded by block-level bounding boxes on this page. The green boxes highlight the included claim, 463.4 kg/ha (block 391), labelled by the row title “Average annual replacement lime required (kg/ha)” (block 387) immediately to its left. The red boxes highlight the excluded claim, 4.6 t/ha (block 393), labelled by the row title “Recommended lime application rate for 10 year period (t/ha)” (block 392). Three of the four candidate extractions returned the red-box value as a candidate answer to the query (which asks specifically for the annual rate); the reviewer rejected it with the note “ This number is for 10 years, question clearly asks for annual. ” The numerical relationship between the two values is internal to the calculator ( $463.4~\mathrm{kg/ha}\times 10/1000\approx 4.6~\mathrm{t/ha}$ ), so a value-only ground truth would not have been able to flag the scope-qualifier mismatch; the locator + disposition + reviewer-note schema makes the rejection auditable from the primary evidence.

Figure 8: Task 3 expert-rated vagueness severity (1 = clear, 5 = maximally ambiguous). The dataset skews hard: 61% of items are severity

\geq

Model	Provider	API	Max Out	Cost (in / out)
GPT-5.5	OpenAI	OpenAI API	32,768	$5.00 /$ 30.00
GPT-5.4-mini	OpenAI	OpenAI API	16,384	$0.75 /$ 4.50
Gemini 3.1 Pro	Google	Vertex AI	64,000	$2.00 /$ 12.00
Gemini 3 Flash	Google	Vertex AI	65,536	$0.50 /$ 3.00
Claude Opus 4.6	Anthropic	Vertex AI	128,000	$5.00 /$ 25.00
Claude Sonnet 4.6	Anthropic	Vertex AI	128,000	$3.00 /$ 15.00

Appendix

Table 4: Baseline models evaluated in PCFBench . Costs are approximate list prices per 1 M tokens at time of evaluation (April 2026).

Figure 9: Task 7 predicted versus actual kgCO 2 e on log–log scale for the best with-region model (lowest median

|\text{RE}|

subject to coverage

\geq

150 of 175 items). Diagonal is perfect prediction; shaded bands are

\pm

\times

and

\pm

\times

. Point colour encodes product category.

Figure 10: Per-task headline metric across the three top frontier models (GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.6). Each vertex of the hexagon is one PCFBench task, scaled to

[0,1]

; metric per axis is given in parentheses. Tasks 2 and 3 are read from the non-agentic pinned runs ( pcfbench_triage and pcfbench_mapping_with_context ); the headline numbers in Table 3 use the agentic harness on those two tasks. Other axes match Table 3 exactly.

Model	Precision	Recall	$F_{1}$	Kendall $\tau$	Exact Set
Gemini 3 Flash	0.898	0.754	0.798	$+$ 0.739	0.213
Claude Opus 4.6 †	0.885	0.739	0.782	$+$ 0.652	0.213
DeepSeek v3.2	0.871	0.702	0.744	$+$ 0.637	0.170
Claude Sonnet 4.6	0.846	0.718	0.744	$+$ 0.624	0.149
Gemini 3.1 Pro †	0.916	0.674	0.740	$+$ 0.756	0.160
GPT-5.5 †	0.910	0.668	0.734	$+$ 0.779	0.149

Appendix

Table 5: Task 1 Product Decomposition full results (

n=94

products). Precision, recall, and

F_{1}

are computed over the judge-aligned matches between predicted and expected components. Kendall

\tau

measures mass-ordering agreement on matched pairs (

+1

= perfect agreement,

0

= random,

-1

= reversed). Exact-set match is the fraction of products where every predicted component matches an expected one and vice versa. Best result per metric in bold .

Model	Accuracy	Format %
Claude Sonnet 4.6	0.650	1.000
Claude Haiku 4.5	0.650	1.000
Claude Opus 4.6 †	0.645	1.000
Gemini 3 Flash	0.630	1.000
Gemini 3.1 Pro †	0.610	0.980
DeepSeek v3.2	0.610	1.000

Appendix

Table 6: Task 2 Mapping Triage full results (

n=200

, balanced 100 map / 100 decompose; expert-labelled, see § B.2 ). The score implementation collapses accuracy = precision = recall = F1 to a single match-rate per item; we report the single value as Accuracy. Cost and latency averages are per-item.

Model	Setting	EM	RS	BA
Gemini 3.1 Pro †	Non-agentic	0.881	1.000	1.000
GPT-5.5 †	Non-agentic	0.835	1.000	0.947
Claude Opus 4.6 †	Non-agentic	0.826	0.957	0.947
Claude Sonnet 4.6	Non-agentic	0.807	0.957	0.947
Gemini 3 Flash	Non-agentic	0.806	1.000	0.946
Claude Haiku 4.5	Non-agentic	0.798	0.957	0.895

Appendix

Table 7: Task 3 Background database mapping with-context full results (

n=109

): non-agentic vs agentic harness, plus the Parakeet specialized baseline. EM = exact match; RS = relevant substring; BA = banned substring absent. Parakeet uses a 3-stage paraphrase

\to

retrieve

\to

rerank pipeline [ Balaji et al., 2025 ] with Sonnet 4.6 as the LLM backbone and GTE-Large embeddings for retrieval, re-implemented in this paper against the 2,574-item picklist. † marks reasoning configurations.

Figure 11: Task 3 mapping exact-match versus expert-rated vagueness severity (with-context setting; per-item exact-match grouped by the 1–5 vagueness label assigned at annotation time). All 8 frontier LLMs hold above 0.85 exact-match on the clearest items (severity 1) and degrade roughly monotonically as ambiguity rises; the spread between models widens at severity 4–5, where the dataset is intentionally concentrated (Figure 8 ). Gemini 3.1 Pro retains the lead through the high-severity tail.

Model	Non-agentic	Agentic	$\boldsymbol{\Delta}$
GPT-5.5	0.650	0.675	$+0.03$
GPT-5.4-mini	0.515	0.620	$+0.10$
Claude Opus 4.6	0.650	0.660	$+0.01$
Claude Sonnet 4.6	0.670	0.725	$+0.05$
Claude Haiku 4.5	0.685	0.715	$+0.03$
Gemini 3.1 Pro	0.705	0.705	$+0.00$

Appendix

Table 8: Task 2 (triage) accuracy: non-agentic vs agentic ecoinvent search/inspect harness,

n=200

\Delta

is agentic minus non-agentic.

Model	Non-agentic	Agentic	$\boldsymbol{\Delta}$
GPT-5.5	0.835	0.798	$-0.04$
GPT-5.4-mini	0.679	0.752	$+0.07$
Claude Opus 4.6	0.826	0.853	$+0.03$
Claude Sonnet 4.6	0.807	0.817	$+0.01$
Claude Haiku 4.5	0.798	0.789	$-0.01$
Gemini 3.1 Pro	0.881	0.927	$+0.05$

Appendix

Table 9: Task 3 (mapping with context) exact-match: non-agentic vs agentic harness,

n=109

\Delta

is agentic minus non-agentic.

Model	Task	$P$	$R$	$F_{1}$	Hall.	UC	Counts (m/p/g)
DeepSeek v3.2	Material (T4)	0.385	0.764	0.512	0.615	0.667	42/109/55
Gemini 3 Flash	Material (T4)	0.348	0.727	0.471	0.652	0.727	40/115/55
Gemini 3.1 Pro †	Material (T4)	0.317	0.709	0.438	0.683	0.762	39/123/55
Claude Sonnet 4.6	Material (T4)	0.250	0.891	0.390	0.750	0.909	49/196/55
Claude Haiku 4.5	Material (T4)	0.245	0.691	0.362	0.755	0.762	38/155/55
GPT-5.5 †	Material (T4)	0.195	0.764	0.311	0.805	0.727	42/215/55

Appendix

Table 10: Tasks 4–5 Physical parameter extraction full results. Material (T4) split: 22 items, 55 ground-truth claims. Energy (T5) split: 14 items, 34 ground-truth claims (36 documents total across the two splits).

P

R

F_{1}

aggregate matched / predicted / ground-truth across all items per split. Hall. = hallucination rate (

1-P

). UC = per-item mean unit-correctness diagnostic (

1

if any predicted unit overlaps any GT unit,

0

otherwise; this is not a headline metric because a model can score high on UC while still missing most claims). Counts are matched / predicted / GT. Best per column in bold (for Hall., lowest is best). All models at 100% format compliance. † marks reasoning configurations.

Model	Setting	Med. $\|$ RE $\|$	$\leq 2\times$	$\leq 5\times$	Mean RE
GPT-5.5 †	Name only	0.412	0.691	0.943	$+$ 0.613
Claude Opus 4.6 †	Name only	0.421	0.686	0.943	$+$ 0.708
Gemini 3.1 Pro †	Name only	0.425	0.680	0.943	$+$ 0.668
Gemini 3 Flash	Name only	0.433	0.686	0.931	$+$ 0.587
Claude Sonnet 4.6	Name only	0.490	0.623	0.897	$+$ 0.972
Claude Haiku 4.5	Name only	0.552	0.571	0.926	$+$ 1.275

Appendix

Table 11: Task 7 total kgCO 2 e prediction full results (

n=175

EPDs). RE

=(\hat{y}-y)/y

\leq F\times

uses the symmetric log-ratio criterion

|\log_{2}(\hat{y}/y)|<\log_{2}F

(i.e.,

1/F<\hat{y}/y<F

); see § F.7 . Positive mean RE indicates systematic overestimation. Best per metric per setting in bold (for Med.

|

|

and Mean RE, smallest absolute value is best). † marks reasoning configurations.

Figure 12: Per-model ghost-component rate on the compositional Task 7 pipeline (BOM entries the model lists with zero mass and which therefore contribute nothing to the sum). Row order matches Figure 3 (a)–(c).

47

–

86\%

of runs include at least one ghost across the eight models; this is the single largest mass-balance failure mode and a likely contributor to the under-prediction tail of compositional outputs visible in panel (b).

Figure 13: Reliability + sharpness diagrams for mapping (panel a; the model sees the material name plus a free-text description, which is the headline “with-context” setting from Table 3 ) and triage single-shot (panel b; the model sees the market name+description plus the parent material context). Top sub-panels: empirical accuracy per confidence bin (bold blue line, markers sized by bin population) versus the perfect-calibration diagonal. Faint blue curves underneath are the same diagram computed per-model (one line per baseline; bins with

n<3

items dropped to suppress single-item swings). The per-model curves cluster around the pooled curve and share its shape relative to the diagonal: under-confident in mapping and over-confident in triage across all eight models and both agentic and non-agentic ablation settings, even though the models differ substantially in headline accuracy: the calibration trend is a property of the task, not of any individual model. Bottom sub-panels: count of predictions per bin. Predictions are pooled across the eight baseline models so per-bin sample sizes support a readable curve; ECE is computed over the same pooled set.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

AI-CARE: Carbon-Aware Reporting Evaluation Metric for AI Models
Feb 17, 2026KC Santosh, Srikanth Baride, Rodrigue RizkCarbon-Aware Computing Energy-Efficient ML
USD Artificial Intelligence Research, Department of Computer Science University of South Dakota Vermillion, SD 57069, USA
Trading Engagement for Sustainability: Carbon-Aware Re-ranking for E-commerce Recommendations
Jun 3, 2026Noah Lund Syrdal, Anders Vestrum, Jorgen BerghLearning to Rank Recommender Systems
University of California, Berkeley
AgentDecarbonizer: Carbon-Aware Execution for AI Agents
Aug 20, 2026Leyi Yan, Shuangning Li, Sihang LiuCarbon-Aware Computing LLM Agent Workflow Optimization
University of Waterloo, Canada · University of Chicago, USA

Feb 17, 2026J/K move · Enter open · S save

AI-CARE: Carbon-Aware Reporting Evaluation Metric for AI Models

KC Santosh, Srikanth Baride, Rodrigue Rizk

USD Artificial Intelligence Research, Department of Computer Science University of South Dakota Vermillion, SD 57069, USA

Carbon-Aware Computing Energy-Efficient ML Multi-Objective Optimization

Open paper AskPDF arXiv

As machine learning (ML) continues its rapid expansion, the environmental cost of model training and inference has become a critical societal concern. Existing benchmarks overwhelmingly focus on standard performance metrics such as accuracy, BLEU, or mAP, while largely ignoring energy consumption and carbon emissions. This single-objective evaluation paradigm is increasingly misaligned with the practical requirements of large-scale deployment, particularly in energy-constrained environments such as mobile devices, developing regions, and climate-aware enterprises. In this paper, we propose AI-CARE, an evaluation tool for reporting energy consumption, and carbon emissions of ML models. In addition, we introduce the carbon-performance tradeoff curve, an interpretable tool that visualizes the Pareto frontier between performance and carbon cost. We demonstrate, through theoretical analysis and empirical validation on representative ML workloads, that carbon-aware benchmarking changes the relative ranking of models and encourages architectures that are simultaneously accurate and environmentally responsible. Our proposal aims to shift the research community toward transparent, multi-objective evaluation and align ML progress with global sustainability goals. The tool and documentation are available at https://github.com/USD-AI-ResearchLab/ai-care.

Abstract

LLM Evaluation

Explore similar work

AI-CARE: Carbon-Aware Reporting Evaluation Metric for AI Models

Trading Engagement for Sustainability: Carbon-Aware Re-ranking for E-commerce Recommendations

AgentDecarbonizer: Carbon-Aware Execution for AI Agents