cs.AIFeb 19, 2026
SaveToward Trustworthy Evaluation of Sustainability Rating Methodologies: A Human-AI Collaborative Framework for Benchmark Dataset Construction
Organizations: Columbia University, USA · Case Western Reserve University, USA · Hong Kong University of Science and Technology, Hong Kong · NVIDIA, USA ·
Abstract
Sustainability or ESG rating agencies use company disclosures and external data to produce scores or ratings that assess the environmental, social, and governance performance of a company. However, sustainability ratings across agencies for a single company vary widely, limiting their comparability, credibility, and relevance to decision-making. To harmonize the rating results, we propose adopting a universal human-AI collaboration framework to generate trustworthy benchmark datasets for evaluating sustainability rating methodologies. The framework comprises two complementary parts: STRIDE (Sustainability Trust Rating & Integrity Data Equation) provides principled criteria and a scoring system that guide the construction of firm-level benchmark datasets using large language models (LLMs), and SR-Delta, a discrepancy-analysis procedural framework that surfaces insights for potential adjustments. The framework enables scalable and comparable assessment of sustainability rating methodologies. We call on the broader AI community to adopt AI-powered approaches to strengthen and advance sustainability rating methodologies that support and enforce urgent sustainability agendas.
Figures & tables
Figure 1: STRIDE framework for trustworthy sustainability rating benchmark datasets. Trust is modeled as a function of credibility, reliability, and human–AI intimacy (positive contributors), and self-serving purpose (negative contributor).
Figure 2: STRIDE trust formulation and component decomposition.
Figure 3: Overview of the STRIDE-guided discrepancy analysis framework. Rating outcomes generated using an existing sustainability rating methodology (A) are compared against ratings produced using STRIDE-guided benchmark data to identify systematic divergences.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The Fortune Global 500 comprises firms with diversity across geographic regions and industry sectors.
| Level | Description | Frameworks and Standards |
|---|---|---|
| Global | International standards | Global Reporting Initiative (GRI); Task Force on Climate-related Financial Disclosures (TCFD); IFRS Sustainability Disclosure Standards (ISSB); UN Sustainable Development Goals (UN SDGs); |
| National | Country-specific regulations | UK Sustainability Reporting Standards; K-ESG Guidelines (Korea); SGX Core ESG Metrics (Singapore); Corporate Sustainability Disclosure Standards (China) |
| Local | Sub-national standards | Implementation Guidance on Climate Information Disclosure under the Hong Kong Exchange Environmental, Social and Governance (ESG) Framework; Beijing Ecological Environment Status Report |
| Industry | Sector-specific standards | Criteria for the Sustainability Assessment of Network Equipment for the Global Electronics |
Table 1: Sustainability Reporting and Disclosure Frameworks Across Four Levels
Figure 5: In our case study, metrics across all levels are consolidated and de-duplicated using LLMs. We recommend incorporating humans in the loop as suggested in the dashed box.
| Criteria | Definition |
|---|---|
| Data Properties | It characterizes the structural, textual, and visual attributes of a document. These metrics serve as proxies for reporting depth, information richness, and disclosure quality. |
| Company Diversity | It represents core organizational characteristics, such as scale, workforce size, and maturity. It captures structural heterogeneity across firms that shapes reporting capacity, disclosure practices, and sustainability performance comparability. |
| Domain Attributes | It represents the extent to which a company’s sustainability reporting engages with environmental, social, and governance domains. It captures variation in topical focus, narrative depth, and standards alignment that shapes disclosure quality. |
Table 2: Conceptual Definitions of Core Dataset Criteria
| Criteria | Metric Name | Metric Definition |
| Data Properties | page_count | Total number of pages in the sustainability report. |
| word_count | Total number of words in the report text. | |
| sentence_count | Total number of sentences extracted from the report. | |
| avg_sentence_length | Average number of words per sentence. | |
| file_size_mb | File size of the report in megabytes (MB). | |
| image_count | Total number of figures, charts, or visual elements. |
Table 3: Dataset Criteria and Metric Definitions
Figure 6: The largest divergence occurs in adopted sustainability frameworks and industry alignment. At a sample size of 300, their Jensen–Shannon Divergence drops to 0.013 and 0.012, indicating distributional stability and sample representativeness.
Figure 7: The graph illustrates, by way of example, the aggregate minimum deviation from the population for the selected samples.
Figure 8: The figure illustrates the end-to-end workflow for metric selection, report selection and sampling for annotation, and modeling. Steps (1–5) consolidate and de-duplicate sustainability metrics across global, sub-national, local, industry, and company levels. Steps (6–10) define report selection, sampling strategy, and human ground-truth annotation. Steps (11–15) integrate annotated datasets with large language models for scalable population, iterative refinement, and data evaluation.
Figure 9: Human–Machine Role Allocation and Oversight Framework
| MSCI Results: The Executive Pay Disclosure scored -1.2. |
|---|
| STRIDE-Guided Extraction Result: An extracted GRI-aligned metric "Has the company failed to disclose specific pay totals for its top executives, including the CEO?". The answer in our dataset is No, which means that -1.2 does not apply |
| Reference Definition from MSCI Methodology (Page 9) : |
| Has the company failed to disclose specific pay totals for its top executives, including the CEO? |
| Flagged if yes. |
| This is the most basic test in this area, and as such, carries considerable weight when invoked, as it must serve in the place of several of the other metrics normally applied in this area. Disclosure must include, at minimum, pay for all executive members of the board of directors (or management board for companies with a two-tier board structure) on an individualized basis or, in cases where there are no executive members, the CEO. The individualized disclosure should include separate information on the amount of each of the following (where such a component is provided): salary, short-term incentives, long-term incentives, pensions, benefits, one-off payments (such as recruitment or retention awards). |
| Analysis: There are two plausible interpretations of how the scoring is applied: 1. Disclosure of an aggregate total would be considered sufficient, even if detailed breakdowns are missing. 2. Points be deducted whenever information is not provided for each compensation category even the total disclosed. |
| MSCI Results: Chemical Safety scored 6.1 |
|---|
| STRIDE-Guided Extraction Result: |
| Luxshare discloses compliance with major chemical regulations (RoHS, REACH), internal EHS management systems, supplier chemical audits, and incoming material inspections. However, the company does not disclose any chemical phase-out roadmap, named priority hazardous substances, quantitative substitution targets, public restricted substance lists (RSL/MRSL), consumer-facing chemical transparency, or time-bound elimination commitments. No supplier non-compliance rates or year-over-year chemical risk reduction metrics are reported. |
| Reference Definition from MSCI Methodology (Page 7) : |
| "MSCI evaluates companies’ exposure to and management of chemical safety risks, including efforts to reduce or eliminate hazardous substances in products and supply chains. Strong management is characterized by robust chemical management systems, supplier controls, substitution strategies, disclosure of restricted substance lists, and proactive phase-out programs aligned with leading frameworks" |
| Analysis: Luxshare demonstrates baseline regulatory compliance and internal chemical controls, which support a mid-range score. However, the absence of disclosed substitution strategies, named priority chemicals, phase-out timelines, public RSL/MRSL, and quantitative performance indicators indicates that Luxshare does not meet MSCI’s criteria for “advanced” or “robust” chemical management. The current MSCI score (6.1) appears to reward compliance-level practices rather than best-practice leadership, creating a mismatch between disclosed evidence and scoring outcomes. |
| Conclusion: |
More in this topic
Synthetic Benchmark Generation
Synthetic benchmark generation creates controlled, verifiable evaluation tasks and datasets to test models across diverse settings.
See all 287 papers