Organizations: Columbia University, USA · Case Western Reserve University, USA · Hong Kong University of Science and Technology, Hong Kong · NVIDIA, USA · Informatica Inc., USA · Uniphore, USA
Sustainability or ESG rating agencies use company disclosures and external data to produce scores or ratings that assess the environmental, social, and governance performance of a company. However, sustainability ratings across agencies for a single company vary widely, limiting their comparability, credibility, and relevance to decision-making. To harmonize the rating results, we propose adopting a universal human-AI collaboration framework to generate trustworthy benchmark datasets for evaluating sustainability rating methodologies. The framework comprises two complementary parts: STRIDE (Sustainability Trust Rating & Integrity Data Equation) provides principled criteria and a scoring system that guide the construction of firm-level benchmark datasets using large language models (LLMs), and SR-Delta, a discrepancy-analysis procedural framework that surfaces insights for potential adjustments. The framework enables scalable and comparable assessment of sustainability rating methodologies. We call on the broader AI community to adopt AI-powered approaches to strengthen and advance sustainability rating methodologies that support and enforce urgent sustainability agendas.
Figures & tables
Figure 1: STRIDE framework for trustworthy sustainability rating benchmark datasets. Trust is modeled as a function of credibility, reliability, and human–AI intimacy (positive contributors), and self-serving purpose (negative contributor).
Figure 2: STRIDE trust formulation and component decomposition.
Figure 3: Overview of the STRIDE-guided discrepancy analysis framework. Rating outcomes generated using an existing sustainability rating methodology (A) are compared against ratings produced using STRIDE-guided benchmark data to identify systematic divergences.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The Fortune Global 500 comprises firms with diversity across geographic regions and industry sectors.
Level
Description
Frameworks and Standards
Global
International standards
Global Reporting Initiative (GRI); Task Force on Climate-related Financial Disclosures (TCFD); IFRS Sustainability Disclosure Standards (ISSB); UN Sustainable Development Goals (UN SDGs);
Implementation Guidance on Climate Information Disclosure under the Hong Kong Exchange Environmental, Social and Governance (ESG) Framework; Beijing Ecological Environment Status Report
Industry
Sector-specific standards
Criteria for the Sustainability Assessment of Network Equipment for the Global Electronics
Appendix
Table 1: Sustainability Reporting and Disclosure Frameworks Across Four Levels
Figure 5: In our case study, metrics across all levels are consolidated and de-duplicated using LLMs. We recommend incorporating humans in the loop as suggested in the dashed box.
Criteria
Definition
Data Properties
It characterizes the structural, textual, and visual attributes of a document. These metrics serve as proxies for reporting depth, information richness, and disclosure quality.
Company Diversity
It represents core organizational characteristics, such as scale, workforce size, and maturity. It captures structural heterogeneity across firms that shapes reporting capacity, disclosure practices, and sustainability performance comparability.
Domain Attributes
It represents the extent to which a company’s sustainability reporting engages with environmental, social, and governance domains. It captures variation in topical focus, narrative depth, and standards alignment that shapes disclosure quality.
Appendix
Table 2: Conceptual Definitions of Core Dataset Criteria
Criteria
Metric Name
Metric Definition
Data Properties
page_count
Total number of pages in the sustainability report.
word_count
Total number of words in the report text.
sentence_count
Total number of sentences extracted from the report.
avg_sentence_length
Average number of words per sentence.
file_size_mb
File size of the report in megabytes (MB).
image_count
Total number of figures, charts, or visual elements.
Appendix
Table 3: Dataset Criteria and Metric Definitions
Figure 6: The largest divergence occurs in adopted sustainability frameworks and industry alignment. At a sample size of 300, their Jensen–Shannon Divergence drops to 0.013 and 0.012, indicating distributional stability and sample representativeness.
Figure 7: The graph illustrates, by way of example, the aggregate minimum deviation from the population for the selected samples.
Figure 8: The figure illustrates the end-to-end workflow for metric selection, report selection and sampling for annotation, and modeling. Steps (1–5) consolidate and de-duplicate sustainability metrics across global, sub-national, local, industry, and company levels. Steps (6–10) define report selection, sampling strategy, and human ground-truth annotation. Steps (11–15) integrate annotated datasets with large language models for scalable population, iterative refinement, and data evaluation.
Figure 9: Human–Machine Role Allocation and Oversight Framework
MSCI Results: The Executive Pay Disclosure scored -1.2.
STRIDE-Guided Extraction Result: An extracted GRI-aligned metric "Has the company failed to disclose specific pay totals for its top executives, including the CEO?". The answer in our dataset is No, which means that -1.2 does not apply
Reference Definition from MSCI Methodology (Page 9) :
Has the company failed to disclose specific pay totals for its top executives, including the CEO?
Flagged if yes.
This is the most basic test in this area, and as such, carries considerable weight when invoked, as it must serve in the place of several of the other metrics normally applied in this area. Disclosure must include, at minimum, pay for all executive members of the board of directors (or management board for companies with a two-tier board structure) on an individualized basis or, in cases where there are no executive members, the CEO. The individualized disclosure should include separate information on the amount of each of the following (where such a component is provided): salary, short-term incentives, long-term incentives, pensions, benefits, one-off payments (such as recruitment or retention awards).
Analysis: There are two plausible interpretations of how the scoring is applied: 1. Disclosure of an aggregate total would be considered sufficient, even if detailed breakdowns are missing. 2. Points be deducted whenever information is not provided for each compensation category even the total disclosed.
Appendix
Table 13
MSCI Results: Chemical Safety scored 6.1
STRIDE-Guided Extraction Result:
Luxshare discloses compliance with major chemical regulations (RoHS, REACH), internal EHS management systems, supplier chemical audits, and incoming material inspections. However, the company does not disclose any chemical phase-out roadmap, named priority hazardous substances, quantitative substitution targets, public restricted substance lists (RSL/MRSL), consumer-facing chemical transparency, or time-bound elimination commitments. No supplier non-compliance rates or year-over-year chemical risk reduction metrics are reported.
Reference Definition from MSCI Methodology (Page 7) :
"MSCI evaluates companies’ exposure to and management of chemical safety risks, including efforts to reduce or eliminate hazardous substances in products and supply chains. Strong management is characterized by robust chemical management systems, supplier controls, substitution strategies, disclosure of restricted substance lists, and proactive phase-out programs aligned with leading frameworks"
Analysis: Luxshare demonstrates baseline regulatory compliance and internal chemical controls, which support a mid-range score. However, the absence of disclosed substitution strategies, named priority chemicals, phase-out timelines, public RSL/MRSL, and quantitative performance indicators indicates that Luxshare does not meet MSCI’s criteria for “advanced” or “robust” chemical management. The current MSCI score (6.1) appears to reward compliance-level practices rather than best-practice leadership, creating a mismatch between disclosed evidence and scoring outcomes.
As corporate responsibility increasingly incorporates environmental, social, and governance (ESG) criteria, ESG reporting is becoming a legal requirement in many regions and a key channel for documenting sustainability practices and assessing firms' long-term and ethical performance. However, the length and complexity of ESG disclosures make them difficult to interpret and automate the analysis reliably. To support scalable and trustworthy analysis, this paper introduces ESG-Bench, a benchmark dataset for ESG report understanding and hallucination mitigation in large language models (LLMs). ESG-Bench contains human-annotated question-answer (QA) pairs grounded in real-world ESG report contexts, with fine-grained labels indicating whether model outputs are factually supported or hallucinated. Framing ESG report analysis as a QA task with verifiability constraints enables systematic evaluation of LLMs' ability to extract and reason over ESG content and provides a new use case: mitigating hallucinations in socially sensitive, compliance-critical settings. We design task-specific Chain-of-Thought (CoT) prompting strategies and fine-tune multiple state-of-the-art LLMs on ESG-Bench using CoT-annotated rationales. Our experiments show that these CoT-based methods substantially outperform standard prompting and direct fine-tuning in reducing hallucinations, and that the gains transfer to existing QA benchmarks beyond the ESG domain.
Siqi Sun, Ben Peng Wu, Mali Jin +3
School of Computer Science, University of Sheffield, Sheffield, UK
The increasing reliance on Large Language Models (LLMs) across diverse sectors highlights the need for robust domain-specific and language-specific evaluation datasets; however, the collection of such datasets is challenging due to privacy concerns, regulatory restrictions, and the time cost for manual creation. Existing automated benchmarking methods are often limited by relying on pre-existing data, poor scalability, single-domain focus, and lack of multilingual support. We present STELLAR-E - a fully automated system to generate high-quality synthetic datasets of custom size, using minimal human inputs without depending on existing datasets. The system is structured in two stages: (1) We modify the TGRT Self-Instruct framework to create a synthetic data engine that enables controllable, custom synthetic dataset generation, and (2) an evaluation pipeline incorporating statistical and LLM-based metrics to assess the applicability of the synthetic dataset for LLM-based application evaluations. The synthetic datasets reach an average difference of +5.7% in terms of LLM-as-a-judge scores against existing language-specific benchmarks, demonstrating comparable quality for comprehensive assessment of big and small LLMs. While real datasets remain slightly more challenging for LLMs especially for smaller models, this work establishes a scalable and domain-adaptable benchmarking framework that supports fair evaluation of LLM applications, offering a faster alternative to manual approaches and enabling high-efficiency automated quality assurance cycles.
Alessio Sordo, Lingxiao Du, Meeka-Hanna Lenisa +2
1Deutsche Bank, Berlin Technology Center, Berlin, Germany
The rapid adoption of Large Language Models (LLMs) has spurred interest in automated peer review; however, progress is currently stifled by benchmarks that treat reviewing primarily as a rating prediction task. We argue that the utility of a review lies in its textual justification--its arguments, questions, and critique--rather than a scalar score. To address this, we introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions: Content Faithfulness, Argumentative Alignment, Focus Consistency, Question Constructiveness, and AI-Likelihood. Notably, we propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews, rigorously filtered to remove procedural noise. Extensive experiments demonstrate that while traditional n-gram metrics fail to reflect human preferences, our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy. These findings establish that aligning AI critique focus with human experts is a prerequisite for reliable automated scoring, offering a robust standard for future research.
Bowen Li, Haochen Ma, Yuxin Wang +5
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China