PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation
Authors: Krishna Rao, Andrew Dumit, Shaena Ulissi, Jacob Feintzeig, P. James Joyce, Daniel Frank, Steven Watson, Jonathan Glidden, Gizem Ilayda Dinc, Travis M. Kwee
AI systems are being deployed on high-stakes, domain-specific workflows that demand correctness not just in the final output, but at every intermediate step. One such workflow is estimating a product carbon footprint (PCF), the greenhouse-gas emissions attributable to a physical product. AI agents are increasingly being used to generate PCFs, but existing evaluations score either total emissions (hiding error sources and cancelling mistakes) or sub-tasks in isolation (missing compositional interactions). We introduce PCFBench, the first benchmark to carve PCF modeling into independently-evaluable tasks that require decomposition, retrieval, ontology matching, and numerical extraction. It comprises 614 expert-labelled items across six tasks. Together they probe reasoning under under-specification, conflicting context, and numerical constraints. Across eight frontier LLMs from four providers, no single model dominates. Although the strongest models estimate total product emissions within 2 times of declared totals on 77% of products, this rate drops to 37-58% when the PCF is generated step by step, with only 45-75% obeying mass conservation. These failures undermine the transparency practitioners need to compare products and drive decarbonization. We release the dataset and evaluation harness to support targeted progress.
Figures & tables
Figure 1: Decomposed PCF creation. Orange brackets denote tasks covered by PCFBench ; grey is deterministic.
Task 1
Task 2
Task 3
Task 4
Task 5
Task 7
Decompose
Triage
Map
Material extraction
Energy extraction
Total kgCO 2 e
Babbitt et al. 2020 [ Babbitt et al., 2020 ]
✓
✗
✗
✗
✗
✗
Parakeet [ Balaji et al., 2025 ]
✗
✗
✓
✗
✗
✗
Sustain-LLaMA [ Kumar et al., 2025 ]
✗
✗
✗
✓
✓
✗
Carbon Catalogue [ Meinrenken et al., 2022 ]
✗
✗
✗
✗
✗
✓
Eco-Amazon [ Spillo et al., 2026 ]
✗
✗
✗
✗
✗
✓
Table 1: Data coverage of PCFBench versus prior datasets. ✓ = task-relevant ground-truth is publicly available; ✗ = no.
Table 2: PCFBench dataset overview and per-task evaluation metrics. Task 6 (multiply and sum) is deterministic and not evaluated. Exact metric definitions in Appendix F .
Figure 2: PCFBench dataset overview. (a) Category coverage : product categories across the six evaluable tasks. Number and shading denote claim/item counts; colour encodes log(count+1) . Furniture, Construction, Vehicles, Electricity-and-fuels, and Services are bucketed into “Other” (Appendix C ). (b) kgCO 2 e distribution : log-scale histogram of kgCO 2 e per kg across the 175 EPDs in Task 7.
Model
Task 1
Task 2
Task 3
Task 4
Task 5
Task 7
Task 7
Decompose
Triage
Map
Materials
Energy
Direct
Compositional
F1
Acc.
EM
F1
F1
≤2×
≤2×
GPT-5.5
0.73 ± 0.02
0.68 ± 0.03
0.80 ± 0.04
0.31 ± 0.07
0.29 ± 0.06
0.77 ± 0.03
0.55 ± 0.04
GPT-5.4-mini
0.70 ± 0.03
0.62 ± 0.03
0.75 ± 0.04
0.29 ± 0.08
0.48 ± 0.10
0.60 ± 0.04
0.50 ± 0.04
Claude Opus 4.6
0.78 ± 0.02
0.66 ± 0.03
0.85 ± 0.03
0.27 ± 0.09
0.29 ± 0.05
0.73 ± 0.03
0.58 ± 0.04
Claude Sonnet 4.6
0.74 ± 0.02
0.72 ± 0.03
0.82 ± 0.04
0.39 ± 0.08
0.39 ± 0.06
0.63 ± 0.04
0.50 ± 0.04
Table 3: PCFBench headline results. Task 1 F1 , Task 2 accuracy, Task 3 exact-match (with-context detail in Appendix F.4 ), Task 4–5 claim- F1 , Task 7 within- 2× of declared kgCO 2 e; exact metric definitions in Appendix F . Tasks 2 and 3 report scores from the agentic harness (single-shot variants in Appendix H.5 ). Direct Task 7: directly predicts the kgCO 2 e of the EPD. Compositional Task 7: an agent assembles kgCO 2 e compositionally by chaining PCFBench ’s task agents on the same EPDs (pipeline spec in Appendix F.8 ). Both modes receive the same input: the product’s name and description. ± values are bootstrap standard errors from 1,000 item-level resamples (protocol in Appendix F.9 ). Parakeet ‡ is a specialized baseline targeting mapping only.
Figure 3: Baseline model performance on PCFBench . (a) Task 7 kgCO 2 e within- 2× rate: direct (one-call) vs compositional (Tasks 1–5 chained, summed via deterministic ecoinvent EFs). Both modes start from product name and description; the compositional pipeline derives composition through Task 1. (b) Per-model relative-error distribution; upper half-violin = direct, lower = compositional. (c) Mass-conservation violation rate (compositional only). (d) Task 2 triage and Task 3 mapping: non-agentic vs agentic accuracy. (e) Accuracy vs increasing context across Tasks 2, 3, 7. (f) Accuracy vs. cost. y -axis is the mean of each model’s within-task min–max-normalised headline scores: for each of the six tasks (T1, T2, T3, T4, T5, T7) we rescale the eight models’ headline metric to [0,1] via (v−vmin)/(vmax−vmin) across the eight models, then average a model’s six rescaled scores. Error bars are the across-task standard error of that mean. x -axis is mean USD spend per benchmark item.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Task 1 BOM size distribution across the 94 products.
Figure 5: Tasks 4–5 dataset characterisation. Left: OCR’d source-document length per item, in thousands of characters (log scale). Median 30.5k, P90 98.1k chars. Centre: number of verbatim evidence quotes per ground-truth claim (mean 2.6, max 8, 227 total). Right: ground-truth value scale per unit family (top-8 by claim count, log scale). Material claims (%, mass%, kg/kg, g/kg) cluster within ∼ two orders of magnitude; energy claims (MJ/kg, kWh/kg) span ∼ three.
Figure 6: Tasks 4 and 5 unit-family distribution across the 55 material and 34 energy ground-truth claims. Cross-family energy units canonicalise to MJ/kg via documented conversion factors at score time.
Figure 7: Annotation example for Tasks 4–5. The full source page is page 6 of Primary Industries and Regions South Australia (2017) [PIRSA] , a one-page Farm Facts calculator that projects an annual liming rate forward to a ten-year amortised application. Both candidate claims are grounded by block-level bounding boxes on this page. The green boxes highlight the included claim, 463.4 kg/ha (block 391), labelled by the row title “Average annual replacement lime required (kg/ha)” (block 387) immediately to its left. The red boxes highlight the excluded claim, 4.6 t/ha (block 393), labelled by the row title “Recommended lime application rate for 10 year period (t/ha)” (block 392). Three of the four candidate extractions returned the red-box value as a candidate answer to the query (which asks specifically for the annual rate); the reviewer rejected it with the note “ This number is for 10 years, question clearly asks for annual. ” The numerical relationship between the two values is internal to the calculator ( 463.4kg/ha×10/1000≈4.6t/ha ), so a value-only ground truth would not have been able to flag the scope-qualifier mismatch; the locator + disposition + reviewer-note schema makes the rejection auditable from the primary evidence.
Figure 8: Task 3 expert-rated vagueness severity (1 = clear, 5 = maximally ambiguous). The dataset skews hard: 61% of items are severity ≥ 4.
Model
Provider
API
Max Out
Cost (in / out)
GPT-5.5
OpenAI
OpenAI API
32,768
5.00/30.00
GPT-5.4-mini
OpenAI
OpenAI API
16,384
0.75/4.50
Gemini 3.1 Pro
Google
Vertex AI
64,000
2.00/12.00
Gemini 3 Flash
Google
Vertex AI
65,536
0.50/3.00
Claude Opus 4.6
Anthropic
Vertex AI
128,000
5.00/25.00
Claude Sonnet 4.6
Anthropic
Vertex AI
128,000
3.00/15.00
Appendix
Table 4: Baseline models evaluated in PCFBench . Costs are approximate list prices per 1 M tokens at time of evaluation (April 2026).
Figure 9: Task 7 predicted versus actual kgCO 2 e on log–log scale for the best with-region model (lowest median ∣RE∣ subject to coverage ≥ 150 of 175 items). Diagonal is perfect prediction; shaded bands are ± 2 × and ± 5 × . Point colour encodes product category.
Figure 10: Per-task headline metric across the three top frontier models (GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.6). Each vertex of the hexagon is one PCFBench task, scaled to [0,1] ; metric per axis is given in parentheses. Tasks 2 and 3 are read from the non-agentic pinned runs ( pcfbench_triage and pcfbench_mapping_with_context ); the headline numbers in Table 3 use the agentic harness on those two tasks. Other axes match Table 3 exactly.
Model
Precision
Recall
F1
Kendall τ
Exact Set
Gemini 3 Flash
0.898
0.754
0.798
+ 0.739
0.213
Claude Opus 4.6 †
0.885
0.739
0.782
+ 0.652
0.213
DeepSeek v3.2
0.871
0.702
0.744
+ 0.637
0.170
Claude Sonnet 4.6
0.846
0.718
0.744
+ 0.624
0.149
Gemini 3.1 Pro †
0.916
0.674
0.740
+ 0.756
0.160
GPT-5.5 †
0.910
0.668
0.734
+ 0.779
0.149
Appendix
Table 5: Task 1 Product Decomposition full results ( n=94 products). Precision, recall, and F1 are computed over the judge-aligned matches between predicted and expected components. Kendall τ measures mass-ordering agreement on matched pairs ( +1 = perfect agreement, 0 = random, −1 = reversed). Exact-set match is the fraction of products where every predicted component matches an expected one and vice versa. Best result per metric in bold .
Model
Accuracy
Format %
Claude Sonnet 4.6
0.650
1.000
Claude Haiku 4.5
0.650
1.000
Claude Opus 4.6 †
0.645
1.000
Gemini 3 Flash
0.630
1.000
Gemini 3.1 Pro †
0.610
0.980
DeepSeek v3.2
0.610
1.000
Appendix
Table 6: Task 2 Mapping Triage full results ( n=200 , balanced 100 map / 100 decompose; expert-labelled, see § B.2 ). The score implementation collapses accuracy = precision = recall = F1 to a single match-rate per item; we report the single value as Accuracy. Cost and latency averages are per-item.
Model
Setting
EM
RS
BA
Gemini 3.1 Pro †
Non-agentic
0.881
1.000
1.000
GPT-5.5 †
Non-agentic
0.835
1.000
0.947
Claude Opus 4.6 †
Non-agentic
0.826
0.957
0.947
Claude Sonnet 4.6
Non-agentic
0.807
0.957
0.947
Gemini 3 Flash
Non-agentic
0.806
1.000
0.946
Claude Haiku 4.5
Non-agentic
0.798
0.957
0.895
Appendix
Table 7: Task 3 Background database mapping with-context full results ( n=109 ): non-agentic vs agentic harness, plus the Parakeet specialized baseline. EM = exact match; RS = relevant substring; BA = banned substring absent. Parakeet uses a 3-stage paraphrase → retrieve → rerank pipeline [ Balaji et al., 2025 ] with Sonnet 4.6 as the LLM backbone and GTE-Large embeddings for retrieval, re-implemented in this paper against the 2,574-item picklist. † marks reasoning configurations.
Figure 11: Task 3 mapping exact-match versus expert-rated vagueness severity (with-context setting; per-item exact-match grouped by the 1–5 vagueness label assigned at annotation time). All 8 frontier LLMs hold above 0.85 exact-match on the clearest items (severity 1) and degrade roughly monotonically as ambiguity rises; the spread between models widens at severity 4–5, where the dataset is intentionally concentrated (Figure 8 ). Gemini 3.1 Pro retains the lead through the high-severity tail.
Model
Non-agentic
Agentic
Δ
GPT-5.5
0.650
0.675
+0.03
GPT-5.4-mini
0.515
0.620
+0.10
Claude Opus 4.6
0.650
0.660
+0.01
Claude Sonnet 4.6
0.670
0.725
+0.05
Claude Haiku 4.5
0.685
0.715
+0.03
Gemini 3.1 Pro
0.705
0.705
+0.00
Appendix
Table 8: Task 2 (triage) accuracy: non-agentic vs agentic ecoinvent search/inspect harness, n=200 . Δ is agentic minus non-agentic.
Model
Non-agentic
Agentic
Δ
GPT-5.5
0.835
0.798
−0.04
GPT-5.4-mini
0.679
0.752
+0.07
Claude Opus 4.6
0.826
0.853
+0.03
Claude Sonnet 4.6
0.807
0.817
+0.01
Claude Haiku 4.5
0.798
0.789
−0.01
Gemini 3.1 Pro
0.881
0.927
+0.05
Appendix
Table 9: Task 3 (mapping with context) exact-match: non-agentic vs agentic harness, n=109 . Δ is agentic minus non-agentic.
Model
Task
P
R
F1
Hall.
UC
Counts (m/p/g)
DeepSeek v3.2
Material (T4)
0.385
0.764
0.512
0.615
0.667
42/109/55
Gemini 3 Flash
Material (T4)
0.348
0.727
0.471
0.652
0.727
40/115/55
Gemini 3.1 Pro †
Material (T4)
0.317
0.709
0.438
0.683
0.762
39/123/55
Claude Sonnet 4.6
Material (T4)
0.250
0.891
0.390
0.750
0.909
49/196/55
Claude Haiku 4.5
Material (T4)
0.245
0.691
0.362
0.755
0.762
38/155/55
GPT-5.5 †
Material (T4)
0.195
0.764
0.311
0.805
0.727
42/215/55
Appendix
Table 10: Tasks 4–5 Physical parameter extraction full results. Material (T4) split: 22 items, 55 ground-truth claims. Energy (T5) split: 14 items, 34 ground-truth claims (36 documents total across the two splits). P , R , F1 aggregate matched / predicted / ground-truth across all items per split. Hall. = hallucination rate ( 1−P ). UC = per-item mean unit-correctness diagnostic ( 1 if any predicted unit overlaps any GT unit, 0 otherwise; this is not a headline metric because a model can score high on UC while still missing most claims). Counts are matched / predicted / GT. Best per column in bold (for Hall., lowest is best). All models at 100% format compliance. † marks reasoning configurations.
Model
Setting
Med. ∣ RE ∣
≤2×
≤5×
Mean RE
GPT-5.5 †
Name only
0.412
0.691
0.943
+ 0.613
Claude Opus 4.6 †
Name only
0.421
0.686
0.943
+ 0.708
Gemini 3.1 Pro †
Name only
0.425
0.680
0.943
+ 0.668
Gemini 3 Flash
Name only
0.433
0.686
0.931
+ 0.587
Claude Sonnet 4.6
Name only
0.490
0.623
0.897
+ 0.972
Claude Haiku 4.5
Name only
0.552
0.571
0.926
+ 1.275
Appendix
Table 11: Task 7 total kgCO 2 e prediction full results ( n=175 EPDs). RE =(y^−y)/y . ≤F× uses the symmetric log-ratio criterion ∣log2(y^/y)∣<log2F (i.e., 1/F<y^/y<F ); see § F.7 . Positive mean RE indicates systematic overestimation. Best per metric per setting in bold (for Med. ∣ RE ∣ and Mean RE, smallest absolute value is best). † marks reasoning configurations.
Figure 12: Per-model ghost-component rate on the compositional Task 7 pipeline (BOM entries the model lists with zero mass and which therefore contribute nothing to the sum). Row order matches Figure 3 (a)–(c). 47 – 86% of runs include at least one ghost across the eight models; this is the single largest mass-balance failure mode and a likely contributor to the under-prediction tail of compositional outputs visible in panel (b).
Figure 13: Reliability + sharpness diagrams for mapping (panel a; the model sees the material name plus a free-text description, which is the headline “with-context” setting from Table 3 ) and triage single-shot (panel b; the model sees the market name+description plus the parent material context). Top sub-panels: empirical accuracy per confidence bin (bold blue line, markers sized by bin population) versus the perfect-calibration diagonal. Faint blue curves underneath are the same diagram computed per-model (one line per baseline; bins with n<3 items dropped to suppress single-item swings). The per-model curves cluster around the pooled curve and share its shape relative to the diagonal: under-confident in mapping and over-confident in triage across all eight models and both agentic and non-agentic ablation settings, even though the models differ substantially in headline accuracy: the calibration trend is a property of the task, not of any individual model. Bottom sub-panels: count of predictions per bin. Predictions are pooled across the eight baseline models so per-bin sample sizes support a readable curve; ECE is computed over the same pooled set.
As machine learning (ML) continues its rapid expansion, the environmental cost of model training and inference has become a critical societal concern. Existing benchmarks overwhelmingly focus on standard performance metrics such as accuracy, BLEU, or mAP, while largely ignoring energy consumption and carbon emissions. This single-objective evaluation paradigm is increasingly misaligned with the practical requirements of large-scale deployment, particularly in energy-constrained environments such as mobile devices, developing regions, and climate-aware enterprises. In this paper, we propose AI-CARE, an evaluation tool for reporting energy consumption, and carbon emissions of ML models. In addition, we introduce the carbon-performance tradeoff curve, an interpretable tool that visualizes the Pareto frontier between performance and carbon cost. We demonstrate, through theoretical analysis and empirical validation on representative ML workloads, that carbon-aware benchmarking changes the relative ranking of models and encourages architectures that are simultaneously accurate and environmentally responsible. Our proposal aims to shift the research community toward transparent, multi-objective evaluation and align ML progress with global sustainability goals. The tool and documentation are available at https://github.com/USD-AI-ResearchLab/ai-care.
KC Santosh, Srikanth Baride, Rodrigue Rizk
USD Artificial Intelligence Research, Department of Computer Science University of South Dakota Vermillion, SD 57069, USA
E-commerce recommender systems strongly influence which products users consider and purchase, yet sustainability signals such as Product Carbon Footprint (PCF) are almost never available at catalog scale. We study carbon-aware product recommendation in the realistic setting where PCF labels are missing for most items and must be inferred. We first estimate product-level carbon footprints via a retrieval-augmented PCF estimation pipeline that transfers supervision from the Carbon Catalogue, a small set of life-cycle-assessed products, to a large unlabeled e-commerce catalog using semantic similarity search, few-shot LLM prompting, and a nearest-neighbour fallback. We then apply a carbon-aware post-hoc re-ranking strategy on top of relevance scores produced by three established recommendation models: BPR, NeuMF, and LightGCN. The method trades off predicted user-item engagement against estimated carbon footprint through a single tunable parameter, lambda. In this offline study, engagement is operationalized through Amazon review interactions, which serve as implicit feedback and as a proxy for user interest or purchase behavior. We evaluate the framework on the Amazon Reviews dataset across three product categories: Home and Kitchen, Sports and Outdoors, and Electronics. By sweeping lambda, we construct Pareto frontiers that characterize the achievable engagement and carbon trade-off for each model and category. Substantial carbon reductions are achievable at minimal engagement cost across all models and categories. However, the available carbon headroom varies by model and category, underscoring the importance of model choice and domain context.
AI agents extend large language models from single prompt-response interactions to long-running, goaldirected workflows that issue many model calls, invoke tools, and interact with external environments. These workflows enable tasks such as software repair, data analysis, and experiment management, but their repeated model invocations can incur substantial carbon emissions. This paper characterizes the carbon emissions of OpenClaw agent workloads using WildClawBench, and shows that emissions depend on token consumption, context cache reuse, and the carbon intensity of the grid. Our characterization identifies deadline flexibility as an opportunity for carbon-aware execution: agent tasks can wait for lower-carbon-intensity periods or shift to lower-carbon grids. However, doing so requires handling uncertain execution time for temporal shifting and cached context recomputation during spatial shifting. We present AgentDecarbonizer, a carbon optimizer for AI agents that runs alongside OpenClaw. Given a task prompt and user-specified deadline, AgentDecarbonizer conservatively estimates task duration and selects deadline-feasible execution schedules, while accounting for cache recomputation overhead during spatial shifting. Evaluated on WildClawBench workloads with 60 agent tasks across four grids, AgentDecarbonizer reduces carbon emissions by up to 57.9 % compared with a carbon-agnostic baseline and by up to 37.5 % compared with a baseline that selects the carbon-optimal grid at task start time.
Leyi Yan, Shuangning Li, Sihang Liu
University of Waterloo, Canada · University of Chicago, USA