METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models

Pengfeng Li; Chen Huang; Chaoqun Hao; Hongyao Chen; Xiao-Yong Wei; Wenqiang Lei; See-Kiong Ng

cs.CLApr 13, 2026

METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models

Authors: Pengfeng Li, Chen Huang, Chaoqun Hao, Hongyao Chen, Xiao-Yong Wei, Wenqiang Lei, See-Kiong Ng

Organizations: College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore

Abstract

Contextual causal reasoning is a critical yet challenging capability for Large Language Models (LLMs). Existing benchmarks, however, often evaluate this skill in fragmented settings, failing to ensure context consistency or cover the full causal hierarchy. To address this, we pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder under a unified context setting. Our extensive evaluation of various LLMs reveals a significant decline in proficiency as tasks ascend the causal hierarchy. To diagnose this degradation, we conduct a deep mechanistic analysis via both error pattern identification and internal information flow tracing. Our analysis reveals two primary failure modes: (1) LLMs are susceptible to distraction by causally irrelevant but factually correct information at lower level of causality; and (2) as tasks ascend the causal hierarchy, faithfulness to the provided context degrades, leading to a reduced performance. We belive our work advances our understanding of the mechanisms behind LLM contextual causal reasoning and establishes a critical foundation for future research. Our code and dataset are available at https://github.com/SCUNLP/METER .

Figures & tables

Figure 1: Illustration of Contextual Causal Reasoning.

Benchmark	Reasoning Paradigm	Causal Discovery	Intervention	Counter- factual	Unified Context	Mechanistic Analysis
ExpliCa Miliani et al. (2025)	Commonsense	✓	✗	✗	✗	✗
CRASS Frohberg and Binder (2022)	Commonsense	✗	✗	✓	✗	✗
CalQuest Lasheras and Pinheiro (2025)	Commonsense	✓	✓	✓	✗	✗
CLADDER Jin et al. (2023)	Formal	✓	✓	✓	✗	✗
CausalBench Wang (2024)	Formal	✓	✓	✓	✗	✗
WIKIWHY Ho et al. (2023)	Contextual	✓	✗	✗	✗	✗

Table 1: Comparison of various benchmarks. Unified Context highlights the contextual inconsistency of current benchmarks. Mechanistic Analysis reveals their failure to understand internal reasoning mechanisms of LLMs.

	Level 1	Level 2	Level 3
# of Contexts	4,145	4,145	4,145
# of Questions	4,145	4,145	4,145
# of Options per Question	5	5	5
Avg. # Tokens per Context	228.91	228.91	228.91
Avg. # Tokens per Question	18.77	26.78	26.16
Avg. # Tokens per Option	17.62	20.28	20.69

Table 2: Data statistics across different causal levels.

Category	Definition
Contradictory Statement	Options conflicting with facts in the context or premises in the question.
Unfounded Statement	Options containing information that is not stated in or cannot be inferred from the context.
Causal Reversal	Options inverting the cause-and-effect direction established in the context.
Irrelevant Fact	Options that are grounded in the context with no causal link to the question.

Table 3: Definitions of the four distractor categories.

Model	Causal Discovery				Intervention				Counterfactual
Model	ZS	CoT	FS	FS+C	ZS	CoT	FS	FS+C	ZS	CoT	FS	FS+C
GPT-4o	87.92	86.46	87.01	87.94	77.96	75.14	78.61	79.45	67.42	66.18	68.99	68.67
Gemini3-Flash	91.02	89.52	91.29	90.55	78.09	78.83	81.17	81.21	70.51	71.08	73.78	72.44
Qwen3-Next (Instruct)	89.56	85.83	85.98	88.46	75.13	76.40	74.20	78.68	64.47	66.29	66.32	68.62
Llama-3.3-70B-Instruct	87.24	85.15	88.40	86.76	78.17	79.33	80.87	77.79	62.08	67.36	70.78	69.55
GPT-5	92.96	–	92.93	–	82.17	–	83.80	–	72.14	–	73.99	–

Table 4: Overall performance across three causal levels with different prompting strategies. ZS : Zero-shot, CoT : Zero-shot CoT, FS : Few-shot, FS+C : Few-shot CoT. Bold denotes the best result across all models. Missing values (–) indicate strategies not applicable to reasoning-optimized models.

Error Type	Gemini3-Flash (%)			Qwen3-4B (%)
	Discovery	Intervention	Counterfactual	Discovery	Intervention	Counterfactual
	(N=424)	(N=953)	(N=1395)	(N=536)	(N=2046)	(N=2489)
Irrelevant Fact	45.56	21.40	16.49	55.41	26.87	24.15
Unfounded Statement	28.89	45.94	50.54	22.20	39.43	36.77
Contradictory Statement	12.00	18.05	26.28	5.69	20.96	33.87
Causal Reversal	13.56	14.60	6.69	16.70	12.74	5.22

Table 5: Error analysis (error percentage) across different causal levels under the zero-shot setting. The total number of incorrect samples

N

is provided in parentheses. Refer to Appendix C.3 for more results.

Panel of Figure 2 — Figure 2: information flow significance at different layers.

Metric	Discovery		Intervention		Counterfactual
Metric	Cor.	Err.	Cor.	Err.	Cor.	Err.
$\bm{S_{E\to O}}$	0.1690	0.1247	0.1144	0.0936	0.1095	0.0988
$\bm{S_{N\to O}}$	0.0945	0.1262	0.0858	0.0858	0.0805	0.0884
$\bm{S_{Q\to O}}$	0.2508	0.2163	0.2657	0.2378	0.2666	0.2281
$\bm{S_{O\to T}}$	0.4685	0.4941	0.5039	0.5457	0.5071	0.5452
$\bm{S_{rest}}$	0.0173	0.0387	0.0302	0.0371	0.0363	0.0395

Table 6: Layer-averaged significance scores for Cor rect vs. Err oneous predictions across causal levels.

Paradigm	Example Input	Model Reasoning Process & Answer
Contextual (Evidence-driven)	Context: In 1991, Godflesh embarked on their first North American tour. While most shows were successful, the schedule faced early setbacks. Specifically, Godflesh missed the first concert due to permit issues , though they made it to the second date in Brooklyn. Despite technical failures with their drum machine later, the rest of the tour concluded with positive reception. Question: Why did the band Godflesh miss their first concert?	Reasoning: I need to find the specific reason for missing the first concert in the provided text. Scanning the passage, I found the sentence: “ Godflesh missed the first concert due to permit issues .” This explicitly states the cause. Answer: The band Godflesh ran into permit issues.
Commonsense (Knowledge-driven)	Premise: The man fell unconscious. What was the cause of this? Alternative 1: The assailant struck the man on the head. Alternative 2: The assailant took the man’s wallet.	Reasoning: I need to identify which event typically causes unconsciousness based on common sense. Alternative 1 describes a physical blow to the head, which causes trauma and is a medically plausible cause of unconsciousness. Alternative 2 describes a theft, which does not directly cause one to lose consciousness. Answer: Alternative 1.
Formal (Rule-driven)	Premise: Imagine a self-contained, hypothetical world with only the following conditions, and without any unmentioned factors or causal relationships: Husband has a direct effect on wife and alarm clock. Wife has a direct effect on alarm clock. The overall probability of alarm set by husband is 77%. For husbands that don’t set the alarm, the probability of ringing alarm is 26%. For husbands that set the alarm, the probability of ringing alarm is 76%. Question: Is ringing alarm more likely than silent alarm overall?	Reasoning: 1. Define Variables: $X$ =Husband, $Y$ =Alarm. 2. Identify structure: $X\to Y$ and $X\to\text{Wife}\to Y$ . 3. Apply Total Probability Theorem for $P(Y=1)$ : $P(Y)=P(Y\|X)P(X)+P(Y\|\neg X)P(\neg X)$ $P(Y)$ $=0.76(0.77)+0.26(1-0.77)$ $P(Y)$ $=0.5852+0.0598\approx 0.645$ 4. Compare with random chance (0.5): $0.645>0.5$ , so ringing is more likely. Answer: Yes.

Table 7: Comparison of causal reasoning paradigms.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Type: Trigger-based Dataset (e.g., MECI)

Context: The engine thrust reverser doors deployed, and the pilot decreased flaps from 40 ∘ to 15 ∘ . The landing gear remained locked in the down position. Six seconds before impact, when the aircraft was 4000 feet from the runway threshold, the aircraft climbed, then banked steeply to the left from a height of 300 to 400 feet, and crashed to the left of the runway. The aircraft was destroyed by impact and fire. The crash investigation was conducted by the Aviation Safety Investigation Division of Transport Canada…

Cause Trigger: impact

Effect Trigger: destroyed

Type: Span-based Dataset (e.g., WIKIWHY)

Context: Even though most ornithocheiromorphs didn’t have a cranial crest like the closely related pteranodontids, there were some exceptions, this included Caulkicephalus and Ludodactylus. Caulkicephalus had a rounded snout , very similar to that of Ornithocheirus and Anhanguera, and therefore it is placed within either Anhangueridae or Ornithocheiridae , depending on the author. Caulkicephalus was also a large pterosaur, with wingspan estimates of around 5 meters (16 ft).

Cause Span: Caulkicephalus has a rounded snout.

Appendix

Table 8: Data examples from the source datasets.

Causal Level	Question Template
Causal Discovery	Asking for Cause Asking for Effect $\bullet$ Why does {EVENT} happen? $\bullet$ What is the result of {EVENT}? $\bullet$ What causes {EVENT}? $\bullet$ What does {EVENT} lead to? $\bullet$ What is the reason behind {EVENT}? $\bullet$ What is the effect of {EVENT}? $\bullet$ What leads to {EVENT}? $\bullet$ What effect does {EVENT} have?
Intervention	What will happen if {INTERVENTION}?
	What change will occur when {INTERVENTION CONDITION}?
	What will be the effect if {INTERVENTION CONDITION}?
	What will be the consequences of {INTERVENTION CONDITION}?
Counterfactual	What would happen if {COUNTERFACTUAL CONDITION}?

Appendix

Table 9: Templates used for generating questions across three causal reasoning levels.

Model & Method	Causal Discovery (%)				Intervention (%)				Counterfactual (%)
Model & Method	Con	Unf	Ctr	Rev	Con	Unf	Ctr	Rev	Con	Unf	Ctr	Rev
Gemini3-Flash
Zero-shot CoT	61.93	24.77	2.42	10.88	39.27	44.66	9.20	6.87	34.28	38.55	23.38	3.79
Few-shot	61.87	25.42	3.34	9.36	38.81	39.05	13.63	8.51	34.46	43.05	4.34	5.22
Few-shot CoT	61.45	27.11	2.71	8.73	43.41	39.02	8.16	9.41	38.15	40.00	17.73	4.12
Gemini3-Pro

Appendix

Table 10: Detailed error distribution analysis for Gemini3-Flash , Gemini3-Pro , and Qwen3-4B under various prompting strategies. The values represent the percentage (%) of each error type within the incorrect predictions. Con : Contextual (Irrelevant Fact), Unf : Unfounded, Ctr : Contradictory, Rev : Causal Reversal.

Metric	Causal Discovery		Intervention		Counterfactual
Metric	Correct	Error	Correct	Error	Correct	Error
Qwen3-8B
$\bm{S_{E\to O}}$	0.1682	0.1058	0.1056	0.0853	0.1095	0.0845
$\bm{S_{N\to O}}$	0.0856	0.1006	0.0785	0.0801	0.0798	0.0784
$\bm{S_{Q\to O}}$	0.2048	0.2008	0.2548	0.2230	0.2448	0.2158
$\bm{S_{O\to T}}$	0.5243	0.5564	0.5329	0.5723	0.5354	0.5801

Appendix

Table 11: Comparison of layer-averaged significance scores between correct and incorrect predictions for Qwen3-8B and Llama-3.2-3B .

Panel of Figure 3 — Figure 3: Layer-wise information flow dynamics for Qwen3-8B (Top Row) and Llama-3.2-3B (Bottom Row) across three causal reasoning levels. The plots illustrate the aggregation trends of saliency scores, contrasting the patterns between correct and incorrect predictions.

Panel of Figure 4 — Figure 4: Results of the Attention Masking experiment on Qwen3-4B . (a) Masking evidence-to-option attention in shallow layers causes a significant drop in Causal Discovery accuracy, confirming reliance on evidence aggregation. (b) Masking the same flow in deep layers results in no performance change, indicating evidence processing is completed in early layers.

Metric	Causal Discovery		Intervention		Counterfactual
Metric	Evidence	No Evidence	Evidence	No Evidence	Evidence	No Evidence
Qwen3-4B
$\bm{S_{E\to O}}$	0.1481	0.1422	0.1080	0.0968	0.1066	0.1003
$\bm{S_{N\to O}}$	0.1034	0.0957	0.0852	0.0863	0.0916	0.0872
$\bm{S_{Q\to O}}$	0.2066	0.2312	0.2204	0.2425	0.2145	0.2336
$\bm{S_{O\to T}}$	0.5132	0.5031	0.5558	0.5421	0.5494	0.5401

Appendix

Table 12: Impact of explicitly incorporating evidence spans on information flow dynamics. Values represent the saliency scores under two settings: with explicitly provided evidence ( Evidence ) and without it ( No Evidence ).

Model	Causal Discovery		Intervention		Counterfactual
Model	Evidence	No Evidence	Evidence	No Evidence	Evidence	No Evidence
Qwen3-4B	84.80	82.67	55.20	53.40	46.40	45.06
Qwen3-8B	88.00	85.40	65.85	63.71	56.40	53.80
Llama-3.2-3B	75.00	72.60	57.83	57.40	44.60	42.03

Appendix

Table 13: Performance comparison (Accuracy) between the original setting ( No Evidence ) and the setting with explicitly incorporated evidence spans ( Evidence ).

Error Type	Qwen3-4B			Qwen3-8B			Llama-3.2-3B
Error Type	Disc.	Inter.	Count.	Disc.	Inter.	Count.	Disc.	Inter.	Count.
Irrelevant Fact	48 $\to$ 35	69 $\to$ 67	91 $\to$ 87	44 $\to$ 29	76 $\to$ 72	57 $\to$ 56	46 $\to$ 36	27 $\to$ 25	47 $\to$ 36
Unfounded	26 $\to$ 25	95 $\to$ 80	75 $\to$ 73	24 $\to$ 23	61 $\to$ 52	77 $\to$ 66	34 $\to$ 28	88 $\to$ 76	107 $\to$ 95
Contradictory	7 $\to$ 5	47 $\to$ 42	98 $\to$ 95	5 $\to$ 4	31 $\to$ 25	89 $\to$ 85	20 $\to$ 15	72 $\to$ 73	95 $\to$ 97
Causal Reversal	6 $\to$ 11	22 $\to$ 35	11 $\to$ 23	0 $\to$ 4	13 $\to$ 22	8 $\to$ 11	37 $\to$ 46	26 $\to$ 37	41 $\to$ 49

Appendix

Table 14: Shift in error distribution after incorporating evidence spans (format: No Evidence

\to

With Evidence ). Blue values indicate a reduction in errors (improvement), while Red values indicate an increase (degradation). Note the consistent reduction in Irrelevant and Unfounded errors versus the rise in Causal Reversal errors.

Context

Henry VI’s two envoys landed in Cyprus in April or May 1196. Aimery may have adopted the title of king around that time, because Pope Celestin styled him as king already in a letter in December 1196. In the same month, the Pope set up a Roman Catholic archdiocese in Nicosia with three suffragan bishops in Famagusta, Limassol and Paphos. The Greek Orthodox bishops were not expelled, but their property and income was seized by the new Catholic prelates.

Level 1: Causal Discovery

Question : Why did Aimery of Cyprus adopt the title of king around April or May of 1196?

Options :

A. The confiscation of the Greek Orthodox bishops’ property and income by Aimery’s newly installed Catholic prelates later in 1196.

B. He had recently married a Byzantine princess and needed a title that matched her royal rank.

Appendix

Table 15: Data samples from our dataset based on a single context.

Case Study: Causal Discovery with Gemini3-Flash (Few-shot)

Context: Wladimir Klitschko stops Tony Thompson in 6th round to retain world heavyweight titles… The Ukrainian champion dropped Thompson again in the sixth, and referee Sam Williams gave the challenger a standing count before stopping the fight …

Question: What caused referee Sam Williams to stop the fight?

Options:

A (Correct - Root Cause): The fight was stopped due to the challenger’s inability to recover after being knocked down by the champion’s powerful punches in consecutive rounds.

B (Model’s Choice - Procedural Distractor): The fight was stopped because the referee’s decision to give a standing count determined that the challenger was no longer fit to continue.

(Options C, D, and E are omitted for brevity)

Appendix

Table 16: A failure case of Gemini3-Flash .

Event Description Expansion

You are an expert in causal event extraction. Given a paragraph P and two event triggers e1 and e2, your task is to identify and summarize the complete events they represent. The event e1 is the cause of e2.

Definitions:

– Event Trigger: A specific word (typically a verb or action-related noun) that serves as the core or anchor of an event.

Guidelines:

For each trigger, locate it in the paragraph and summarize its corresponding event. Each summary must be:

– Accurate: only what the trigger states, no extra causes/effects.

Appendix

Table 17: The prompt used to expand event triggers into complete event descriptions.

Data De-contamination

You are an expert in causal inference. Your task is to analyze Event A and Event B provided below and determine whether there is a causal relationship between them. Respond with "Yes" if a causal relationship exists, and "No" otherwise.

Event A: <event1> Event B: <event2>

Answer:

Appendix

Table 18: The prompt used to filter out cause-effect pairs that LLMs might have memorized.

Intervention Question and Answer Generation

You are an expert in causal inference, expert at crafting deep, multi-layered causal inquiries. You will be given:

Paragraph : a paragraph describing a causal scenario.

Cause and Effect Events : two events, which are causally connected.

Your task is to instantiate the template below to design a intervention question and give a correct answer based on the given cause and effect events.

The intervention question explores the future causal consequences of introducing a new and specific action (the “intervention”). Crucially, the intervention must be specific and explore a diverse state, not a vague or simple negation. You cannot modify or negate the fixed history of the scenario. The question must be in the future tense grammatically. The answer must be tightly grounded in the Paragraph, serving as a logical extension of the facts and dynamics provided. Ensure the answer is concise, accurate.

Question Template: <question template>

Appendix

Table 19: Prompts used for generating Intervention and Counterfactual questions along with their answers.

Distractor Generation

You are an expert in constructing multiple-choice causal reasoning questions. You will be given:

- Paragraph : a paragraph describing a causal scenario.

- Question : a

<

question_type

>

- Correct Option : the correct option.

Your task is to generate four distinct and misleading incorrect options (distractors), each corresponding to a different fallacy type. For each generated distractor, you must also provide a brief rationale explaining why the option is incorrect yet sufficiently plausible to be misleading.

Distractor Types:

Appendix

Table 20: Prompt used for generating distractors.

Prompting Schemes

Zero-shot : Pick one choice given the context to answer the question. Context :

\langle

Context

\rangle

Question :

\langle

Question

\rangle

Choices : A.

\langle

Choice A

\rangle

\langle

Choice B

\rangle

\langle

Choice C

\rangle

\langle

Choice D

\rangle

\langle

Choice E

\rangle

Answer :

Few-shot : Pick one choice given the context to answer the question. Example: Context : The city library, usually open late, closed two hours early last month. For several days beforehand, the lights in the reading rooms kept flickering. In a council bulletin, officials explained that the building’s electrical system had problems, though some residents blamed budget cuts. Question : Why did the library close earlier than usual last month? Choices : A. The city held a council bulletin. B. The library’s electrical system was malfunctioning. C. Closing early caused the lights to flicker. D. Budget cuts reduced funding for cultural programs. E. The library lights flickered in the evenings. Answer : B Context :

\langle

Context

\rangle

Question :

\langle

Question

\rangle

Choices : A.

\langle

Choice A

\rangle

\langle

Choice B

\rangle

\langle

Choice C

\rangle

\langle

Choice D

\rangle

\langle

Choice E

\rangle

Answer :

Zero-shot CoT : Pick one choice given the context to answer the question. Context :

\langle

Context

\rangle

Question :

\langle

Question

\rangle

Choices : A.

\langle

Choice A

\rangle

\langle

Choice B

\rangle

\langle

Choice C

\rangle

\langle

Choice D

\rangle

\langle

Choice E

\rangle

Let’s think step by step.

Few-shot CoT : Pick one choice given the context to answer the question. Example: Context : The city library, usually open late, closed two hours early last month. For several days beforehand, the lights in the reading rooms kept flickering. In a council bulletin, officials explained that the building’s electrical system had problems, though some residents blamed budget cuts. Question : Why did the library close earlier than usual last month? Choices : A. The city held a council bulletin. B. The library’s electrical system was malfunctioning. C. Closing early caused the lights to flicker. D. Budget cuts reduced funding for cultural programs. E. The library lights flickered in the evenings. Answer : Officials stated the building’s electrical system had problems, which explains the early closure. A is irrelevant: the council bulletin is merely where the explanation appears, not a cause. B is correct: it aligns perfectly with the official explanation. C reverses causality: flickering happened beforehand and is due to electrical issues, not caused by closing early. D is unsupported conjecture (residents’ blame) and is outweighed by the official explanation. E describes a symptom/timing detail, not the reason for closing early. Correct answer : B Context :

\langle

Context

\rangle

Question :

\langle

Question

\rangle

Choices : A.

\langle

Choice A

\rangle

\langle

Choice B

\rangle

\langle

Choice C

\rangle

\langle

Choice D

\rangle

\langle

Choice E

\rangle

Answer :

Appendix

Table 21: Prompting templates used in our experiments across four settings: Zero-shot, Few-shot, Zero-shot CoT, and Few-shot CoT.

Causal Reasoning in Language Models

Causal reasoning in language models studies how models identify cause-and-effect structure, predict interventions, and answer counterfactual questions.

See all 102 papers

Contextual Causality with Large Language Models: A Survey
Sep 18, 2026Yiheng Zhao, Jun Yan, Chengming HuCausal Reasoning in Language Models Causal Interventions in Language Models
Concordia University, Montreal, Canada · Mcgill University, Montreal, Canada
CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
Apr 9, 2024Yu Zhou, Xingyu Wu, Jibin Wu +2LLM Evaluation Causal Discovery
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching
Feb 23, 2026Yuzhe Wang, Yaochen Zhu, Jundong LiLLM Evaluation Causal Reasoning in Language Models
University of Virginia Charlottesville, Virginia, USA

Sep 18, 2026J/K move · Enter open · S save

Contextual Causality with Large Language Models: A Survey

Yiheng Zhao, Jun Yan, Chengming Hu

Concordia University, Montreal, Canada · Mcgill University, Montreal, Canada

Causal Reasoning in Language Models Causal Interventions in Language Models Counterfactual Reasoning in Language Models

Open paper AskPDF arXiv

Understanding contextual causality is critical for large language models (LLMs), as it enables them to accurately identify causal relations in specific situations and support more reliable decision-making. Despite its significance, a systematic exploration of contextual causality with LLMs is still lacking. To fill this gap, we present a comprehensive survey on this topic. In this survey, we first propose a taxonomy of contextual causality, consisting of semantic, intervention, and counterfactual causality, and characterize each category by its core causal question, required model capabilities, representative tasks, and practical uses in causality analysis. We then analyze existing studies and discuss their key limitations. Finally, we examine the gaps between current benchmarks and real-world needs and outline promising directions for future research. Our goal is to clarify the research landscape of contextual causality with LLMs, emphasize its importance, and highlight promising future directions.

Abstract

Causal Reasoning in Language Models

Explore similar work

Contextual Causality with Large Language Models: A Survey

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching