Organizations: College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore
Contextual causal reasoning is a critical yet challenging capability for Large Language Models (LLMs). Existing benchmarks, however, often evaluate this skill in fragmented settings, failing to ensure context consistency or cover the full causal hierarchy. To address this, we pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder under a unified context setting. Our extensive evaluation of various LLMs reveals a significant decline in proficiency as tasks ascend the causal hierarchy. To diagnose this degradation, we conduct a deep mechanistic analysis via both error pattern identification and internal information flow tracing. Our analysis reveals two primary failure modes: (1) LLMs are susceptible to distraction by causally irrelevant but factually correct information at lower level of causality; and (2) as tasks ascend the causal hierarchy, faithfulness to the provided context degrades, leading to a reduced performance. We belive our work advances our understanding of the mechanisms behind LLM contextual causal reasoning and establishes a critical foundation for future research. Our code and dataset are available at https://github.com/SCUNLP/METER .
Figures & tables
Figure 1: Illustration of Contextual Causal Reasoning.
Benchmark
Reasoning Paradigm
Causal Discovery
Intervention
Counter- factual
Unified Context
Mechanistic Analysis
ExpliCa Miliani et al. (2025)
Commonsense
✓
✗
✗
✗
✗
CRASS Frohberg and Binder (2022)
Commonsense
✗
✗
✓
✗
✗
CalQuest Lasheras and Pinheiro (2025)
Commonsense
✓
✓
✓
✗
✗
CLADDER Jin et al. (2023)
Formal
✓
✓
✓
✗
✗
CausalBench Wang (2024)
Formal
✓
✓
✓
✗
✗
WIKIWHY Ho et al. (2023)
Contextual
✓
✗
✗
✗
✗
Table 1: Comparison of various benchmarks. Unified Context highlights the contextual inconsistency of current benchmarks. Mechanistic Analysis reveals their failure to understand internal reasoning mechanisms of LLMs.
Level 1
Level 2
Level 3
# of Contexts
4,145
4,145
4,145
# of Questions
4,145
4,145
4,145
# of Options per Question
5
5
5
Avg. # Tokens per Context
228.91
228.91
228.91
Avg. # Tokens per Question
18.77
26.78
26.16
Avg. # Tokens per Option
17.62
20.28
20.69
Table 2: Data statistics across different causal levels.
Category
Definition
Contradictory Statement
Options conflicting with facts in the context or premises in the question.
Unfounded Statement
Options containing information that is not stated in or cannot be inferred from the context.
Causal Reversal
Options inverting the cause-and-effect direction established in the context.
Irrelevant Fact
Options that are grounded in the context with no causal link to the question.
Table 3: Definitions of the four distractor categories.
Model
Causal Discovery
Intervention
Counterfactual
ZS
CoT
FS
FS+C
ZS
CoT
FS
FS+C
ZS
CoT
FS
FS+C
GPT-4o
87.92
86.46
87.01
87.94
77.96
75.14
78.61
79.45
67.42
66.18
68.99
68.67
Gemini3-Flash
91.02
89.52
91.29
90.55
78.09
78.83
81.17
81.21
70.51
71.08
73.78
72.44
Qwen3-Next (Instruct)
89.56
85.83
85.98
88.46
75.13
76.40
74.20
78.68
64.47
66.29
66.32
68.62
Llama-3.3-70B-Instruct
87.24
85.15
88.40
86.76
78.17
79.33
80.87
77.79
62.08
67.36
70.78
69.55
GPT-5
92.96
–
92.93
–
82.17
–
83.80
–
72.14
–
73.99
–
Table 4: Overall performance across three causal levels with different prompting strategies. ZS : Zero-shot, CoT : Zero-shot CoT, FS : Few-shot, FS+C : Few-shot CoT. Bold denotes the best result across all models. Missing values (–) indicate strategies not applicable to reasoning-optimized models.
Error Type
Gemini3-Flash (%)
Qwen3-4B (%)
Discovery
Intervention
Counterfactual
Discovery
Intervention
Counterfactual
(N=424)
(N=953)
(N=1395)
(N=536)
(N=2046)
(N=2489)
Irrelevant Fact
45.56
21.40
16.49
55.41
26.87
24.15
Unfounded Statement
28.89
45.94
50.54
22.20
39.43
36.77
Contradictory Statement
12.00
18.05
26.28
5.69
20.96
33.87
Causal Reversal
13.56
14.60
6.69
16.70
12.74
5.22
Table 5: Error analysis (error percentage) across different causal levels under the zero-shot setting. The total number of incorrect samples N is provided in parentheses. Refer to Appendix C.3 for more results.
Figure 2: information flow significance at different layers.
Metric
Discovery
Intervention
Counterfactual
Cor.
Err.
Cor.
Err.
Cor.
Err.
SE→O
0.1690
0.1247
0.1144
0.0936
0.1095
0.0988
SN→O
0.0945
0.1262
0.0858
0.0858
0.0805
0.0884
SQ→O
0.2508
0.2163
0.2657
0.2378
0.2666
0.2281
SO→T
0.4685
0.4941
0.5039
0.5457
0.5071
0.5452
Srest
0.0173
0.0387
0.0302
0.0371
0.0363
0.0395
Table 6: Layer-averaged significance scores for Cor rect vs. Err oneous predictions across causal levels.
Paradigm
Example Input
Model Reasoning Process & Answer
Contextual (Evidence-driven)
Context: In 1991, Godflesh embarked on their first North American tour. While most shows were successful, the schedule faced early setbacks. Specifically, Godflesh missed the first concert due to permit issues , though they made it to the second date in Brooklyn. Despite technical failures with their drum machine later, the rest of the tour concluded with positive reception. Question: Why did the band Godflesh miss their first concert?
Reasoning: I need to find the specific reason for missing the first concert in the provided text. Scanning the passage, I found the sentence: “ Godflesh missed the first concert due to permit issues .” This explicitly states the cause. Answer: The band Godflesh ran into permit issues.
Commonsense (Knowledge-driven)
Premise: The man fell unconscious. What was the cause of this? Alternative 1: The assailant struck the man on the head. Alternative 2: The assailant took the man’s wallet.
Reasoning: I need to identify which event typically causes unconsciousness based on common sense. Alternative 1 describes a physical blow to the head, which causes trauma and is a medically plausible cause of unconsciousness. Alternative 2 describes a theft, which does not directly cause one to lose consciousness. Answer: Alternative 1.
Formal (Rule-driven)
Premise: Imagine a self-contained, hypothetical world with only the following conditions, and without any unmentioned factors or causal relationships: Husband has a direct effect on wife and alarm clock. Wife has a direct effect on alarm clock. The overall probability of alarm set by husband is 77%. For husbands that don’t set the alarm, the probability of ringing alarm is 26%. For husbands that set the alarm, the probability of ringing alarm is 76%. Question: Is ringing alarm more likely than silent alarm overall?
Reasoning: 1. Define Variables: X =Husband, Y =Alarm. 2. Identify structure: X→Y and X→Wife→Y . 3. Apply Total Probability Theorem for P(Y=1) : P(Y)=P(Y∣X)P(X)+P(Y∣¬X)P(¬X)P(Y)=0.76(0.77)+0.26(1−0.77)P(Y)=0.5852+0.0598≈0.645 4. Compare with random chance (0.5): 0.645>0.5 , so ringing is more likely. Answer: Yes.
Table 7: Comparison of causal reasoning paradigms.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Type: Trigger-based Dataset (e.g., MECI)
Context: The engine thrust reverser doors deployed, and the pilot decreased flaps from 40 ∘ to 15 ∘ . The landing gear remained locked in the down position. Six seconds before impact, when the aircraft was 4000 feet from the runway threshold, the aircraft climbed, then banked steeply to the left from a height of 300 to 400 feet, and crashed to the left of the runway. The aircraft was destroyed by impact and fire. The crash investigation was conducted by the Aviation Safety Investigation Division of Transport Canada…
Cause Trigger: impact
Effect Trigger: destroyed
Type: Span-based Dataset (e.g., WIKIWHY)
Context: Even though most ornithocheiromorphs didn’t have a cranial crest like the closely related pteranodontids, there were some exceptions, this included Caulkicephalus and Ludodactylus. Caulkicephalus had a rounded snout , very similar to that of Ornithocheirus and Anhanguera, and therefore it is placed within either Anhangueridae or Ornithocheiridae , depending on the author. Caulkicephalus was also a large pterosaur, with wingspan estimates of around 5 meters (16 ft).
Cause Span: Caulkicephalus has a rounded snout.
Appendix
Table 8: Data examples from the source datasets.
Causal Level
Question Template
Causal Discovery
Asking for Cause Asking for Effect ∙ Why does {EVENT} happen? ∙ What is the result of {EVENT}? ∙ What causes {EVENT}? ∙ What does {EVENT} lead to? ∙ What is the reason behind {EVENT}? ∙ What is the effect of {EVENT}? ∙ What leads to {EVENT}? ∙ What effect does {EVENT} have?
Intervention
What will happen if {INTERVENTION}?
What change will occur when {INTERVENTION CONDITION}?
What will be the effect if {INTERVENTION CONDITION}?
What will be the consequences of {INTERVENTION CONDITION}?
Counterfactual
What would happen if {COUNTERFACTUAL CONDITION}?
Appendix
Table 9: Templates used for generating questions across three causal reasoning levels.
Model & Method
Causal Discovery (%)
Intervention (%)
Counterfactual (%)
Con
Unf
Ctr
Rev
Con
Unf
Ctr
Rev
Con
Unf
Ctr
Rev
Gemini3-Flash
Zero-shot CoT
61.93
24.77
2.42
10.88
39.27
44.66
9.20
6.87
34.28
38.55
23.38
3.79
Few-shot
61.87
25.42
3.34
9.36
38.81
39.05
13.63
8.51
34.46
43.05
4.34
5.22
Few-shot CoT
61.45
27.11
2.71
8.73
43.41
39.02
8.16
9.41
38.15
40.00
17.73
4.12
Gemini3-Pro
Appendix
Table 10: Detailed error distribution analysis for Gemini3-Flash , Gemini3-Pro , and Qwen3-4B under various prompting strategies. The values represent the percentage (%) of each error type within the incorrect predictions. Con : Contextual (Irrelevant Fact), Unf : Unfounded, Ctr : Contradictory, Rev : Causal Reversal.
Metric
Causal Discovery
Intervention
Counterfactual
Correct
Error
Correct
Error
Correct
Error
Qwen3-8B
SE→O
0.1682
0.1058
0.1056
0.0853
0.1095
0.0845
SN→O
0.0856
0.1006
0.0785
0.0801
0.0798
0.0784
SQ→O
0.2048
0.2008
0.2548
0.2230
0.2448
0.2158
SO→T
0.5243
0.5564
0.5329
0.5723
0.5354
0.5801
Appendix
Table 11: Comparison of layer-averaged significance scores between correct and incorrect predictions for Qwen3-8B and Llama-3.2-3B .
Figure 3: Layer-wise information flow dynamics for Qwen3-8B (Top Row) and Llama-3.2-3B (Bottom Row) across three causal reasoning levels. The plots illustrate the aggregation trends of saliency scores, contrasting the patterns between correct and incorrect predictions.
Figure 4: Results of the Attention Masking experiment on Qwen3-4B . (a) Masking evidence-to-option attention in shallow layers causes a significant drop in Causal Discovery accuracy, confirming reliance on evidence aggregation. (b) Masking the same flow in deep layers results in no performance change, indicating evidence processing is completed in early layers.
Metric
Causal Discovery
Intervention
Counterfactual
Evidence
No Evidence
Evidence
No Evidence
Evidence
No Evidence
Qwen3-4B
SE→O
0.1481
0.1422
0.1080
0.0968
0.1066
0.1003
SN→O
0.1034
0.0957
0.0852
0.0863
0.0916
0.0872
SQ→O
0.2066
0.2312
0.2204
0.2425
0.2145
0.2336
SO→T
0.5132
0.5031
0.5558
0.5421
0.5494
0.5401
Appendix
Table 12: Impact of explicitly incorporating evidence spans on information flow dynamics. Values represent the saliency scores under two settings: with explicitly provided evidence ( Evidence ) and without it ( No Evidence ).
Model
Causal Discovery
Intervention
Counterfactual
Evidence
No Evidence
Evidence
No Evidence
Evidence
No Evidence
Qwen3-4B
84.80
82.67
55.20
53.40
46.40
45.06
Qwen3-8B
88.00
85.40
65.85
63.71
56.40
53.80
Llama-3.2-3B
75.00
72.60
57.83
57.40
44.60
42.03
Appendix
Table 13: Performance comparison (Accuracy) between the original setting ( No Evidence ) and the setting with explicitly incorporated evidence spans ( Evidence ).
Error Type
Qwen3-4B
Qwen3-8B
Llama-3.2-3B
Disc.
Inter.
Count.
Disc.
Inter.
Count.
Disc.
Inter.
Count.
Irrelevant Fact
48 → 35
69 → 67
91 → 87
44 → 29
76 → 72
57 → 56
46 → 36
27 → 25
47 → 36
Unfounded
26 → 25
95 → 80
75 → 73
24 → 23
61 → 52
77 → 66
34 → 28
88 → 76
107 → 95
Contradictory
7 → 5
47 → 42
98 → 95
5 → 4
31 → 25
89 → 85
20 → 15
72 → 73
95 → 97
Causal Reversal
6 → 11
22 → 35
11 → 23
0 → 4
13 → 22
8 → 11
37 → 46
26 → 37
41 → 49
Appendix
Table 14: Shift in error distribution after incorporating evidence spans (format: No Evidence → With Evidence ). Blue values indicate a reduction in errors (improvement), while Red values indicate an increase (degradation). Note the consistent reduction in Irrelevant and Unfounded errors versus the rise in Causal Reversal errors.
Context
Henry VI’s two envoys landed in Cyprus in April or May 1196. Aimery may have adopted the title of king around that time, because Pope Celestin styled him as king already in a letter in December 1196. In the same month, the Pope set up a Roman Catholic archdiocese in Nicosia with three suffragan bishops in Famagusta, Limassol and Paphos. The Greek Orthodox bishops were not expelled, but their property and income was seized by the new Catholic prelates.
Level 1: Causal Discovery
Question : Why did Aimery of Cyprus adopt the title of king around April or May of 1196?
Options :
A. The confiscation of the Greek Orthodox bishops’ property and income by Aimery’s newly installed Catholic prelates later in 1196.
B. He had recently married a Byzantine princess and needed a title that matched her royal rank.
Appendix
Table 15: Data samples from our dataset based on a single context.
Case Study: Causal Discovery with Gemini3-Flash (Few-shot)
Context: Wladimir Klitschko stops Tony Thompson in 6th round to retain world heavyweight titles… The Ukrainian champion dropped Thompson again in the sixth, and referee Sam Williams gave the challenger a standing count before stopping the fight …
Question: What caused referee Sam Williams to stop the fight?
Options:
A (Correct - Root Cause): The fight was stopped due to the challenger’s inability to recover after being knocked down by the champion’s powerful punches in consecutive rounds.
B (Model’s Choice - Procedural Distractor): The fight was stopped because the referee’s decision to give a standing count determined that the challenger was no longer fit to continue.
(Options C, D, and E are omitted for brevity)
Appendix
Table 16: A failure case of Gemini3-Flash .
Event Description Expansion
You are an expert in causal event extraction. Given a paragraph P and two event triggers e1 and e2, your task is to identify and summarize the complete events they represent. The event e1 is the cause of e2.
Definitions:
– Event Trigger: A specific word (typically a verb or action-related noun) that serves as the core or anchor of an event.
Guidelines:
For each trigger, locate it in the paragraph and summarize its corresponding event. Each summary must be:
– Accurate: only what the trigger states, no extra causes/effects.
Appendix
Table 17: The prompt used to expand event triggers into complete event descriptions.
Data De-contamination
You are an expert in causal inference. Your task is to analyze Event A and Event B provided below and determine whether there is a causal relationship between them. Respond with "Yes" if a causal relationship exists, and "No" otherwise.
Event A: <event1> Event B: <event2>
Answer:
Appendix
Table 18: The prompt used to filter out cause-effect pairs that LLMs might have memorized.
Intervention Question and Answer Generation
You are an expert in causal inference, expert at crafting deep, multi-layered causal inquiries. You will be given:
Paragraph : a paragraph describing a causal scenario.
Cause and Effect Events : two events, which are causally connected.
Your task is to instantiate the template below to design a intervention question and give a correct answer based on the given cause and effect events.
The intervention question explores the future causal consequences of introducing a new and specific action (the “intervention”). Crucially, the intervention must be specific and explore a diverse state, not a vague or simple negation. You cannot modify or negate the fixed history of the scenario. The question must be in the future tense grammatically. The answer must be tightly grounded in the Paragraph, serving as a logical extension of the facts and dynamics provided. Ensure the answer is concise, accurate.
Question Template: <question template>
Appendix
Table 19: Prompts used for generating Intervention and Counterfactual questions along with their answers.
Distractor Generation
You are an expert in constructing multiple-choice causal reasoning questions. You will be given:
- Paragraph : a paragraph describing a causal scenario.
- Question : a < question_type > .
- Correct Option : the correct option.
Your task is to generate four distinct and misleading incorrect options (distractors), each corresponding to a different fallacy type. For each generated distractor, you must also provide a brief rationale explaining why the option is incorrect yet sufficiently plausible to be misleading.
Distractor Types:
Appendix
Table 20: Prompt used for generating distractors.
Prompting Schemes
Zero-shot : Pick one choice given the context to answer the question. Context : ⟨ Context ⟩ Question : ⟨ Question ⟩ Choices : A. ⟨ Choice A ⟩ B. ⟨ Choice B ⟩ C. ⟨ Choice C ⟩ D. ⟨ Choice D ⟩ E. ⟨ Choice E ⟩ Answer :
Few-shot : Pick one choice given the context to answer the question. Example: Context : The city library, usually open late, closed two hours early last month. For several days beforehand, the lights in the reading rooms kept flickering. In a council bulletin, officials explained that the building’s electrical system had problems, though some residents blamed budget cuts. Question : Why did the library close earlier than usual last month? Choices : A. The city held a council bulletin. B. The library’s electrical system was malfunctioning. C. Closing early caused the lights to flicker. D. Budget cuts reduced funding for cultural programs. E. The library lights flickered in the evenings. Answer : B Context : ⟨ Context ⟩ Question : ⟨ Question ⟩ Choices : A. ⟨ Choice A ⟩ B. ⟨ Choice B ⟩ C. ⟨ Choice C ⟩ D. ⟨ Choice D ⟩ E. ⟨ Choice E ⟩ Answer :
Zero-shot CoT : Pick one choice given the context to answer the question. Context : ⟨ Context ⟩ Question : ⟨ Question ⟩ Choices : A. ⟨ Choice A ⟩ B. ⟨ Choice B ⟩ C. ⟨ Choice C ⟩ D. ⟨ Choice D ⟩ E. ⟨ Choice E ⟩ Let’s think step by step.
Few-shot CoT : Pick one choice given the context to answer the question. Example: Context : The city library, usually open late, closed two hours early last month. For several days beforehand, the lights in the reading rooms kept flickering. In a council bulletin, officials explained that the building’s electrical system had problems, though some residents blamed budget cuts. Question : Why did the library close earlier than usual last month? Choices : A. The city held a council bulletin. B. The library’s electrical system was malfunctioning. C. Closing early caused the lights to flicker. D. Budget cuts reduced funding for cultural programs. E. The library lights flickered in the evenings. Answer : Officials stated the building’s electrical system had problems, which explains the early closure. A is irrelevant: the council bulletin is merely where the explanation appears, not a cause. B is correct: it aligns perfectly with the official explanation. C reverses causality: flickering happened beforehand and is due to electrical issues, not caused by closing early. D is unsupported conjecture (residents’ blame) and is outweighed by the official explanation. E describes a symptom/timing detail, not the reason for closing early. Correct answer : B Context : ⟨ Context ⟩ Question : ⟨ Question ⟩ Choices : A. ⟨ Choice A ⟩ B. ⟨ Choice B ⟩ C. ⟨ Choice C ⟩ D. ⟨ Choice D ⟩ E. ⟨ Choice E ⟩ Answer :
Appendix
Table 21: Prompting templates used in our experiments across four settings: Zero-shot, Few-shot, Zero-shot CoT, and Few-shot CoT.
Causal reasoning in language models studies how models identify cause-and-effect structure, predict interventions, and answer counterfactual questions.
Understanding contextual causality is critical for large language models (LLMs), as it enables them to accurately identify causal relations in specific situations and support more reliable decision-making. Despite its significance, a systematic exploration of contextual causality with LLMs is still lacking. To fill this gap, we present a comprehensive survey on this topic. In this survey, we first propose a taxonomy of contextual causality, consisting of semantic, intervention, and counterfactual causality, and characterize each category by its core causal question, required model capabilities, representative tasks, and practical uses in causality analysis. We then analyze existing studies and discuss their key limitations. Finally, we examine the gaps between current benchmarks and real-world needs and outline promising directions for future research. Our goal is to clarify the research landscape of contextual causality with LLMs, emphasize its importance, and highlight promising future directions.
The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual reasoning, as causality reveals the underlying data distribution. However, the lack of a comprehensive benchmark currently limits the evaluation of LLMs' causal learning capabilities. To fill this gap, this paper develops CausalBN-Bench based on data from the causal research community, enabling comparative evaluations of LLMs against traditional causal learning algorithms. To provide a comprehensive investigation, we offer three tasks of varying difficulties, including correlation, causal skeleton, and causality identification. Evaluations of 19 leading LLMs reveal that, while closed-source LLMs show potential for simple causal relationships, they significantly lag behind traditional algorithms on larger-scale networks (>50 nodes). Specifically, LLMs struggle with collider structures but excel at chain structures, especially at long-chain causality analogous to Chain-of-Thought techniques. This supports the current prompt approaches while suggesting directions to enhance LLMs' causal reasoning capability. Furthermore, CausalBN-Bench incorporates background knowledge and training data into prompts to thoroughly unlock LLMs' text-comprehension ability during evaluation, whose findings indicate that, LLM understand causality through semantic associations with distinct entities, rather than directly from contextual information or numerical distributions.
Yu Zhou, Xingyu Wu, Jibin Wu +2
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
As large language models (LLMs) witness increasing deployment in complex, high-stakes decision-making scenarios, it becomes imperative to ground their reasoning in causality rather than spurious correlations. However, strong performance on traditional reasoning benchmarks does not guarantee true causal reasoning ability of LLMs, as high accuracy may still arise from memorizing semantic patterns instead of analyzing the underlying true causal structures. To bridge this critical gap, we propose a new causal reasoning benchmark, CausalFlip, designed to encourage the development of new LLM paradigm or training algorithms that ground LLM reasoning in causality rather than semantic correlation. CausalFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations. Based on this, for each event triple, we construct pairs of semantically similar questions that reuse the same events but yield opposite causal answers, where models that rely heavily on semantic matching are systematically driven toward incorrect predictions. To further probe models' reliance on semantic patterns, we introduce a noisy-prefix evaluation that prepends causally irrelevant text before intermediate causal reasoning steps without altering the underlying causal relations or the logic of the reasoning process. We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought (CoT) supervision, and a proposed internalized causal reasoning approach that aims to mitigate explicit reliance on correlation in the reasoning process. Our results show that explicit CoT can still be misled by spurious semantic correlations, where internalizing reasoning steps yields substantially improved causal grounding, suggesting that it is promising to better elicit the latent causal reasoning capabilities of base LLMs.
Yuzhe Wang, Yaochen Zhu, Jundong Li
University of Virginia Charlottesville, Virginia, USA