Climate change is a major socio-scientific issue shapes public decision-making and policy discussions. As large language models (LLMs) increasingly serve as an interface for accessing climate knowledge, whether existing benchmarks reflect user needs is critical for evaluating LLM in real-world settings. We propose a Proactive Knowledge Behaviors Framework that captures the different human-human and human-AI knowledge seeking and provision behaviors. We further develop a Topic-Intent-Form taxonomy and apply it to analyze climate-related data representing different knowledge behaviors. Our results reveal a substantial mismatch between current benchmarks and real-world user needs, while knowledge interaction patterns between humans and LLMs closely resemble those in human-human interactions. These findings provide actionable guidance for benchmark design, RAG system development, and LLM training. Code is available at https://github.com/OuchengLiu/LLM-Misalign-Climate-Change.
Figures & tables
Figure 1: Our Proactive Knowledge Behaviors Framework. The proactive knowledge behaviors between three key actors are shown as blue arrows and the red arrows reflect our analytical logic.
Figure 2: Pipelines for data annotation in Topic Identification and Question Type Classification.
Figure 3: Topic comparison between Human-to-AI Queries and Human-to-AI Guidance Knowledge. (a) Pairwise topic-distribution similarities across the five datasets; (b) Probabilities of the 10 most diverging topics, i.e., those topics with the highest absolute probability differences under the two groups; (c) Topic-distribution similarities between each dataset and each group; (d) Probability differences of the most diverging topics. The interpretations and presentations of Figures 4 and 5 follow the same logic.
Figure 4: User intents comparison between Human-to-AI Queries and Human-to-AI Guidance Knowledge.
Figure 5: Expected answer forms comparison between Human-to-AI Queries and Human-to-AI Guidance Knowledge.
Figure 6: Question-type differences within topic A1. Atmospheric Science&Climate Processes : Real-World queries to LLM vs LLM QA benchmarks. (a) Top-10 intent differences; (b) Top-10 answer form differences.
Figure 7: Topic comparison between real-world queries to LLMs and questions to humans. (a) Pairwise topic-distribution similarity across four datasets; (b) Topic distributions for the union of Top-10 topics (by share) across the four datasets. The interpretations and presentations of Figures 8 and 9 follow the same logic.
Figure 8: User intent comparison between real-world queries to LLMs and real-world questions to humans.
Figure 9: Expected answer form comparison between real-world queries to LLMs and real-world questions to humans.
Figure 10: Topic comparison between inform-human knowledge and both real-world questions to human and queries to LLMs.
Figure 11: Topic differences: inform-human knowledge vs real-world questions to human. (a) IPCC minus Reddit topic distributions; (b) SciDCC minus Reddit topic distributions. Only shows the union of Top-10 absolute differences.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Final Topic Taxonomy of Climate Change.
Figure 13: Taxonomy of question types, including user intents the expected answer forms. The Intent Taxonomy also contains a “Knowledge” label indicating the type of knowledge required by an LLM for each intent: F for Factual, C for Conceptual, P for Procedural, and M for Metacognitive. Each category also has an z. Others type, which is omitted from the figure.
k=min(B,max(1,⌊∣T∣/10⌋)).
Appendix
Algorithm 1 Iterative Topic Merging
Figure 14: Topic Similarity for All 11 Datasets.
Figure 15: Intent Similarity for All 11 Datasets.
Figure 16: Form Similarity for All 11 Datasets.
Figure 17: Topic Distribution for All 11 Datasets.
Figure 18: Intent Distribution for All 11 Datasets.
Figure 19: Form Distribution for All 11 Datasets.
Figure 20: Topic × Intent cross-analysis.
Figure 21: Topic × Form cross-analysis.
Figure 22: Intent × Form cross-analysis.
Figure 23: Topic Similarity for All 11 Datasets under S1 Merge Setting.
Figure 24: Topic Similarity for All 11 Datasets under S2 Merge Setting.
Figure 25: Topic Similarity for All 11 Datasets under S3 Merge Setting.
Figure 26: Topic Similarity for All 11 Datasets under S4 Merge Setting.
Figure 27: Topic Similarity for All 11 Datasets under S5 Merge Setting.
Figure 28: Topic Similarity for All 11 Datasets under S6 Merge Setting.
Figure 29: Interactive data visualization enabling flexible data selection, sample weighting, and both multidimensional and cross-dimensional analysis. Shown here is a partial cross-analysis between Intent and Form.
Figure 30: Visualization of the topic-merging process, with interactive access to each topic’s explanation, sample count, and randomly sampled instances. Leaf nodes are displayed in purple, while other nodes are shown in gray.
Figure 31: Similarity between the sampled validation subset and the full dataset.
Mitigating climate change requires behaviour change. However, even climate-concerned individuals often hold misperceptions about which actions most reduce carbon emissions. We recruited 1201 climate-concerned individuals to examine whether discussing climate actions with a large language model (LLM) equipped with climate knowledge and prompted to provide personalised responses would foster more accurate perceptions of the impacts of climate actions and increase willingness to adopt feasible, high-impact behaviours. We compared this to having participants run a web search, have a conversation with an unspecialised LLM, and no intervention. The personalised climate LLM was the only condition that led to increased knowledge about the impacts of climate actions and greater intentions to adopt impactful behaviours. While the personalised climate LLM did not outperform a web search in improving understanding of climate action impacts, the ability of LLMs to deliver personalised, actionable guidance may make them more effective at motivating impactful pro-climate behaviour change.
Miriam Remshard, Yara Kyrychenko, Sander van der Linden +4
University of Cambridge Cambridge, UK · Yale University New Haven, CT, USA · Harvard T.H. Chan School of Public Health Boston, MA, USA
Climate research is pivotal for mitigating global environmental crises, yet the accelerating volume of multi-scale datasets and the complexity of analytical tools have created significant bottlenecks, constraining scientific discovery to fragmented and labor-intensive workflows. While the emergence Large Language Models (LLMs) offers a transformative paradigm to scale scientific expertise, existing explorations remain largely confined to simple Question-Answering (Q&A) tasks. These approaches often oversimplify real-world challenges, neglecting the intricate physical constraints and the data-driven nature required in professional climate science.To bridge this gap, we introduce ClimAgent, a general-purpose autonomous framework designed to execute a wide spectrum of research tasks across diverse climate sub-fields. By integrating a unified tool-use environment with rigorous reasoning protocols, ClimAgent transcends simple retrieval to perform end-to-end modeling and analysis. To foster systematic evaluation, we propose ClimaBench, the first comprehensive benchmark for real-world climate discovery. It encompasses challenging problems spanning 5 distinct task categories derived from professional scenarios between 2000 and 2025. Experiments on ClimaBench demonstrate that ClimAgent significantly outperforms state-of-the-art baselines, achieving a 40.21% improvement over original LLM solutions in solution rigorousness and practicality. Our code are available at https://github.com/usail-hkust/ClimAgent.
Hao Wang, Jindong Han, Wei Fan +1
1The Hong Kong University of Science and Technology (Guangzhou) · 3Shandong University · University of Auckland +1
While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions, revealing a substantial Knowledge-Decision Gap (GKD). Existing benchmarks struggle to measure this discrepancy due to limited construct validity, multidimensional entanglement, and distributional biases in LLM-based evaluation. To address these issues, we propose ActTraitBench, a human-grounded evaluation framework for measuring personality consistency in LLMs. Grounded in empirical human data, ActTraitBench establishes one-to-one mappings between psychometric facets and behavioral paradigms and applies Distributional Calibration via Quantile Mapping to reduce distributional mismatch between LLM-judge scores and human responses. Experiments on 14 mainstream LLMs reveal substantial knowledge-decision gaps and show that assigned personas are reflected more consistently in self-reports than in behavioral decisions for most evaluated models. To mitigate this gap, we further introduce the Chain of Cognitive Alignment (CoCA), an inference-time intervention that reduces GKD for 12 of the 13 models with paired results. Code and resources are available at https://github.com/Selina233/ActTraitBench.