Understanding contextual causality is critical for large language models (LLMs), as it enables them to accurately identify causal relations in specific situations and support more reliable decision-making. Despite its significance, a systematic exploration of contextual causality with LLMs is still lacking. To fill this gap, we present a comprehensive survey on this topic. In this survey, we first propose a taxonomy of contextual causality, consisting of semantic, intervention, and counterfactual causality, and characterize each category by its core causal question, required model capabilities, representative tasks, and practical uses in causality analysis. We then analyze existing studies and discuss their key limitations. Finally, we examine the gaps between current benchmarks and real-world needs and outline promising directions for future research. Our goal is to clarify the research landscape of contextual causality with LLMs, emphasize its importance, and highlight promising future directions.
Figures & tables
Table 1: Comparison with methodology-oriented causality surveys. ✓ indicates systematic coverage, indicates coverage of only a few relevant studies, and ✗ indicates no systematic coverage. ( Wang et al., 2024b ) mainly covers counterfactual generation (CG), whereas ours covers broader counterfactual reasoning (CR).
Core Causal Question
Capability
Task
Practical Use
Semantic Causality
What causal information is expressed or implied in context?
Identification
Event Causality Identification
Causal Graph Construction
Causal Judgment
Causal Link Refinement
Causal Attribution
Responsibility Assignment
Causal Detection and Span Extraction
Causal Filtering and Evidence Grounding
Explanation
Causal Explanation Generation
Causal Mechanism Explanation
Table 2: A taxonomy-driven overview of contextual causality with LLMs. The “Core Question” column specifies the causal question addressed by each category. The “Capability” column summarizes the causal ability required of LLMs for each category. The “Tasks” column lists representative tasks corresponding to each capability. The “Practical Use” column describes how each task can be employed in practical causality analysis.
Benchmark
Source
Focus
Semantic Causality
Event Causal Identification
MAVEN-ERE Wang et al. (2022a)
Wiki
Intrinsic
MECI Lai et al. (2022)
Wiki
Intrinsic
ESL Caselli and Vossen (2017)
News
Intrinsic
CTB Mirza et al. (2014)
News
Intrinsic
Table 3: Representative benchmarks for contextual causality. “Focus” indicates whether a benchmark measures causality ability or how much the causality ability of current LLMs improves decision-making for real-world needs. “Intrinsic” denotes the former, while “Extrinsic” denotes the latter. “Wiki”, “W&N”, “Human”, and “Agents” denote Wikipedia-derived data, mixed Wikipedia and news sources, human-constructed based on world or commonsense knowledge, and agent-system traces or execution logs, respectively.
Causal reasoning in language models studies how models identify cause-and-effect structure, predict interventions, and answer counterfactual questions.
Contextual causal reasoning is a critical yet challenging capability for Large Language Models (LLMs). Existing benchmarks, however, often evaluate this skill in fragmented settings, failing to ensure context consistency or cover the full causal hierarchy. To address this, we pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder under a unified context setting. Our extensive evaluation of various LLMs reveals a significant decline in proficiency as tasks ascend the causal hierarchy. To diagnose this degradation, we conduct a deep mechanistic analysis via both error pattern identification and internal information flow tracing. Our analysis reveals two primary failure modes: (1) LLMs are susceptible to distraction by causally irrelevant but factually correct information at lower level of causality; and (2) as tasks ascend the causal hierarchy, faithfulness to the provided context degrades, leading to a reduced performance. We belive our work advances our understanding of the mechanisms behind LLM contextual causal reasoning and establishes a critical foundation for future research. Our code and dataset are available at https://github.com/SCUNLP/METER .
Pengfeng Li, Chen Huang, Chaoqun Hao +4
College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore
The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual reasoning, as causality reveals the underlying data distribution. However, the lack of a comprehensive benchmark currently limits the evaluation of LLMs' causal learning capabilities. To fill this gap, this paper develops CausalBN-Bench based on data from the causal research community, enabling comparative evaluations of LLMs against traditional causal learning algorithms. To provide a comprehensive investigation, we offer three tasks of varying difficulties, including correlation, causal skeleton, and causality identification. Evaluations of 19 leading LLMs reveal that, while closed-source LLMs show potential for simple causal relationships, they significantly lag behind traditional algorithms on larger-scale networks (>50 nodes). Specifically, LLMs struggle with collider structures but excel at chain structures, especially at long-chain causality analogous to Chain-of-Thought techniques. This supports the current prompt approaches while suggesting directions to enhance LLMs' causal reasoning capability. Furthermore, CausalBN-Bench incorporates background knowledge and training data into prompts to thoroughly unlock LLMs' text-comprehension ability during evaluation, whose findings indicate that, LLM understand causality through semantic associations with distinct entities, rather than directly from contextual information or numerical distributions.
Yu Zhou, Xingyu Wu, Jibin Wu +2
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
Large language models (LLMs) are trained on enormous amounts of data and encode knowledge in their parameters. We propose a pipeline to elicit causal relationships from LLMs. Specifically, (i) we sample many documents from LLMs on a given topic, (ii) we extract an event list from from each document, (iii) we group events that appear across documents into canonical events, (iv) we construct a binary indicator vector for each document over canonical events, and (v) we estimate candidate causal graphs using causal discovery methods. Our approach does not guarantee real-world causality. Rather, it provides a framework for presenting the set of causal hypotheses that LLMs can plausibly assume, as an inspectable set of variables and candidate graphs.