Climate change is a major socio-scientific issue shapes public decision-making and policy discussions. As large language models (LLMs) increasingly serve as an interface for accessing climate knowledge, whether existing benchmarks reflect user needs is critical for evaluating LLM in real-world settings. We propose a Proactive Knowledge Behaviors Framework that captures the different human-human and human-AI knowledge seeking and provision behaviors. We further develop a Topic-Intent-Form taxonomy and apply it to analyze climate-related data representing different knowledge behaviors. Our results reveal a substantial mismatch between current benchmarks and real-world user needs, while knowledge interaction patterns between humans and LLMs closely resemble those in human-human interactions. These findings provide actionable guidance for benchmark design, RAG system development, and LLM training. Code is available at https://github.com/OuchengLiu/LLM-Misalign-Climate-Change.
Figures & tables
Figure 1: Our Proactive Knowledge Behaviors Framework. The proactive knowledge behaviors between three key actors are shown as blue arrows and the red arrows reflect our analytical logic.
Figure 2: Pipelines for data annotation in Topic Identification and Question Type Classification.
Figure 3: Topic comparison between Human-to-AI Queries and Human-to-AI Guidance Knowledge. (a) Pairwise topic-distribution similarities across the five datasets; (b) Probabilities of the 10 most diverging topics, i.e., those topics with the highest absolute probability differences under the two groups; (c) Topic-distribution similarities between each dataset and each group; (d) Probability differences of the most diverging topics. The interpretations and presentations of Figures 4 and 5 follow the same logic.
Figure 4: User intents comparison between Human-to-AI Queries and Human-to-AI Guidance Knowledge.
Figure 5: Expected answer forms comparison between Human-to-AI Queries and Human-to-AI Guidance Knowledge.
Figure 6: Question-type differences within topic A1. Atmospheric Science&Climate Processes : Real-World queries to LLM vs LLM QA benchmarks. (a) Top-10 intent differences; (b) Top-10 answer form differences.
Figure 7: Topic comparison between real-world queries to LLMs and questions to humans. (a) Pairwise topic-distribution similarity across four datasets; (b) Topic distributions for the union of Top-10 topics (by share) across the four datasets. The interpretations and presentations of Figures 8 and 9 follow the same logic.
Figure 8: User intent comparison between real-world queries to LLMs and real-world questions to humans.
Figure 9: Expected answer form comparison between real-world queries to LLMs and real-world questions to humans.
Figure 10: Topic comparison between inform-human knowledge and both real-world questions to human and queries to LLMs.
Figure 11: Topic differences: inform-human knowledge vs real-world questions to human. (a) IPCC minus Reddit topic distributions; (b) SciDCC minus Reddit topic distributions. Only shows the union of Top-10 absolute differences.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Final Topic Taxonomy of Climate Change.
Figure 13: Taxonomy of question types, including user intents the expected answer forms. The Intent Taxonomy also contains a “Knowledge” label indicating the type of knowledge required by an LLM for each intent: F for Factual, C for Conceptual, P for Procedural, and M for Metacognitive. Each category also has an z. Others type, which is omitted from the figure.
k=min(B,max(1,⌊∣T∣/10⌋)).
Appendix
Algorithm 1 Iterative Topic Merging
Figure 14: Topic Similarity for All 11 Datasets.
Figure 15: Intent Similarity for All 11 Datasets.
Figure 16: Form Similarity for All 11 Datasets.
Figure 17: Topic Distribution for All 11 Datasets.
Figure 18: Intent Distribution for All 11 Datasets.
Figure 19: Form Distribution for All 11 Datasets.
Figure 20: Topic × Intent cross-analysis.
Figure 21: Topic × Form cross-analysis.
Figure 22: Intent × Form cross-analysis.
Figure 23: Topic Similarity for All 11 Datasets under S1 Merge Setting.
Figure 24: Topic Similarity for All 11 Datasets under S2 Merge Setting.
Figure 25: Topic Similarity for All 11 Datasets under S3 Merge Setting.
Figure 26: Topic Similarity for All 11 Datasets under S4 Merge Setting.
Figure 27: Topic Similarity for All 11 Datasets under S5 Merge Setting.
Figure 28: Topic Similarity for All 11 Datasets under S6 Merge Setting.
Figure 29: Interactive data visualization enabling flexible data selection, sample weighting, and both multidimensional and cross-dimensional analysis. Shown here is a partial cross-analysis between Intent and Form.
Figure 30: Visualization of the topic-merging process, with interactive access to each topic’s explanation, sample count, and randomly sampled instances. Leaf nodes are displayed in purple, while other nodes are shown in gray.
Figure 31: Similarity between the sampled validation subset and the full dataset.