Evaluating Causal Reasoning in LLMs: A Case Study in the Challenges of Procedural Generation

Justin Clarke; Isha Karn; Patrick Murray; KJ Hill; David Jensen

cs.AIOct 28, 2026

Evaluating Causal Reasoning in LLMs: A Case Study in the Challenges of Procedural Generation

Authors: Justin Clarke, Isha Karn, Patrick Murray, KJ Hill, David Jensen

Organizations: University of Massachusetts Amherst

Abstract

Large language models (LLMs) appear to represent and reason about a large range of phenomena. In particular, recent studies have examined whether LLMs can accurately reason about causation. One of the most commonly used benchmarks for evaluating formal causal reasoning in LLMs is CLadder, a framework for procedural generation of benchmark data sets that was released in 2023. Experiments with CLadder appear to indicate that LLMs can perform formal causal reasoning in a large percentage of generated examples. In this paper, we examine what can be reliably inferred when LLMs perform well on test sets produced by CLadder. We identify several examples of superficial and highly predictive patterns that the CLadder generator embeds in data sets it generates. We find that even simple models such as random forests perform well on CLadder-generated data sets after training on a sample of generated data. We also find that the performance of many commercial LLMs on CLadder test sets is extremely brittle and that removing large sections of the text that are critical for formal causal reasoning has almost no effect on performance. Given these results, the most likely hypothesis for the strong performance of state-of-the-art LLMs on CLadder is that they are exploiting superficial patterns learned from exposure to training data from the generator. Any inference that LLMs are exhibiting more complex causal reasoning must rule out this hypothesis.

Figures & tables

Figure 1: This paper examines how and why LLMs perform well on CL adder , a dynamic benchmark for evaluating causal reasoning, and what can be inferred from the results.

Figure 2: Scatter plots of the two key conditional probabilities provided in average treatment effect (ATE) and effect of treatment on the treated (ETT) queries, colored by correct answer. Within each polarity condition, yes and no answers are nearly linearly separable. A model that learns this simple comparison rule can achieve near-perfect accuracy on these query types without any formal causal reasoning.

Figure 3: Model accuracies by query type. The first two rows show accuracy by query type on our generated CL adder test set for a random forest classifier trained on superficial features derived from the query text or BERT embeddings. We report the mean accuracy over 10-fold cross-validation. The third row shows the performance of OLMo-3.1-32B after fine-tuning on CL adder . We also show gpt-5.6-sol with and without reasoning for comparison.

Figure 4: Model accuracy compared to the date of the model’s release. Open circles represent the published ”knowledge cutoff” dates for each model. The results for the text-davinci models come from the original CL adder paper.

Figure 5: Model accuracy under interventions that remove elements of the queries that are critical for formal causal reasoning. A full query-wise breakdown in Appendix A.6 . The 95% Wilson confidence intervals are not visible in this plot due to large sample size.

Figure 6: Model accuracy under interventions that remove elements of the queries that make it impossible to calculate the answer. Full query-wise breakdown in Appendix A.6 . The 95% Wilson confidence intervals are not visible in this plot due to large sample size.

Figure 7: Token ablation results for a set of marginal queries on gpt-5.2. Each row corresponds to a single prompt, and each cell represents one token. Red cells denote tokens whose removal reduces model performance, while green cells denote tokens whose removal increases performance. Numerical tokens (#), percentage tokens (%), and the polarity indicator are always labeled. Otherwise, labels are shown only for tokens exceeding a salience threshold ( $|\Delta|\geq 0.04$ ).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Query type	Count	Rung
Marginal	1200	1
Correlation	1200	1
Collider bias	432	1
Backdoor adjustment	74	2
ATE	1200	2
ETT	1200	2

Appendix

Table 1: Query-type distribution in the generated dataset.

Split	$N$	Acc. (%)	95% CI	$p$ -value
Generated	9,866	94.73	$\pm$ 0.4	—
Easy	10,559	94.16	$\pm$ 0.4	0.074
Hard	10,556	94.49	$\pm$ 0.4	0.443
Non-commonsense	10,240	94.11	$\pm$ 0.5	0.056
Anti-commonsense	10,392	95.20	$\pm$ 0.4	0.127

Appendix

Table 2: Zero-shot GPT-5.5 (medium reasoning) accuracy on our generated dataset ( Generated ) and each CLadder v1 split individually. No split is significantly harder than the generated dataset (

p

-values from two-proportion

z

-tests, Bonferroni-corrected threshold

p<0.0125

, none significant).

Non-LLM baselines
Model	Overall	ATE	MAR	COR	EXP	NDE	DCF	NIE	ETT	COL	BKA
RF	86.8 $\pm$ 0.7	96.8 $\pm$ 1.0	93.8 $\pm$ 1.4	90.4 $\pm$ 1.7	92.2 $\pm$ 1.5	81.2 $\pm$ 2.2	48.6 $\pm$ 3.2	81.5 $\pm$ 2.2	99.0 $\pm$ 0.6	100.0 $\pm$ 0.4	62.2 $\pm$ 10.8
RF+BERT	76.2 $\pm$ 0.8	83.8 $\pm$ 2.1	62.7 $\pm$ 2.7	85.8 $\pm$ 2.0	80.2 $\pm$ 2.3	71.5 $\pm$ 2.6	51.7 $\pm$ 3.2	81.8 $\pm$ 2.2	81.6 $\pm$ 2.2	97.9 $\pm$ 1.4	39.2 $\pm$ 10.9
Open-weight models
Llama-3.1-8B-Instruct	55.7 $\pm$ 1.0	79.5 $\pm$ 2.3	52.2 $\pm$ 2.8	46.3 $\pm$ 2.8	52.8 $\pm$ 2.8	51.2 $\pm$ 2.8	61.0 $\pm$ 3.1	49.0 $\pm$ 2.8	60.8 $\pm$ 2.8	39.8 $\pm$ 4.6	50.0 $\pm$ 11.1
Qwen2.5-3B-Instruct	52.1 $\pm$ 1.0	87.6 $\pm$ 1.9	47.8 $\pm$ 2.8	54.0 $\pm$ 2.8	15.4 $\pm$ 2.0	55.4 $\pm$ 2.8	54.5 $\pm$ 3.1	54.0 $\pm$ 2.8	49.8 $\pm$ 2.8	50.0 $\pm$ 4.7	50.0 $\pm$ 11.1

Appendix

Table 3: Model accuracy on generated Cladder queries (

\%

\pm

Wilson 95 % CI half-width). Bold = best per column. † Back-Adj has

n{=}74

; all other types

n{\geq}432

Figure 8: Model performance as a function of the number of publicly-available CL adder samples the model is exposed to during fine-tuning.

Figure 9: Model performance (per query-type) as a function of the number of publicly-available CL adder samples the model is exposed to during fine-tuning.

Figure 10: Per-query-type model accuracy under interventions that remove elements of the queries that are critical for formal causal reasoning. We show 95% Wilson CI.

Figure 11: Per-query-type model accuracy under interventions that remove elements of the queries that make it impossible to calculate the answer. We show 95% Wilson CI.

Model	Type	Acc. (%)
Simple models ( $n=1200$ )
Random Forest	Classifier	$79.1\pm 2.3$
Commercial models (one-word prompt, $n=1200$ )
GPT-3.5-turbo	Base	$48.6\pm 2.8$
GPT-4.1	Base	$48.8\pm 2.8$
GPT-4o	Base	$52.2\pm 2.8$

Appendix

Table 4: CounterBench Alpha accuracy

\pm

half-width of Wilson 95% CI (chance = 50%). All commercial GPT models perform at or near chance regardless of generation. Fine-tuned open-weight models and the random forest outperform all commercial models.

Figure 12: Token ablation analysis on gpt-5.2 for a random sample of 25 evaluated marginal queries. Same visualization as Figure 7 . Color encoding, label conventions, and salience threshold are identical to the main figure.

Causal Reasoning in Language Models

Causal reasoning in language models studies how models identify cause-and-effect structure, predict interventions, and answer counterfactual questions.

See all 104 papers

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
Apr 9, 2024Yu Zhou, Xingyu Wu, Jibin Wu +2LLM Evaluation Causal Discovery
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching
Feb 23, 2026Yuzhe Wang, Yaochen Zhu, Jundong LiLLM Evaluation Causal Reasoning in Language Models
University of Virginia Charlottesville, Virginia, USA
Towards a Universal Causal Reasoner
May 24, 2026Qirun Dai, Xiao Liu, Jiawei Zhang +3Causal Reasoning in Language Models Causal Reasoning
1The University of Chicago · University of Illinois Urbana-Champaign

Apr 9, 2024J/K move · Enter open · S save

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng+1

Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China

LLM Evaluation Causal Discovery Causal Structure Learning Causal Reasoning in Language Models Causal Reasoning

Open paper AskPDF arXiv

The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual reasoning, as causality reveals the underlying data distribution. However, the lack of a comprehensive benchmark currently limits the evaluation of LLMs' causal learning capabilities. To fill this gap, this paper develops CausalBN-Bench based on data from the causal research community, enabling comparative evaluations of LLMs against traditional causal learning algorithms. To provide a comprehensive investigation, we offer three tasks of varying difficulties, including correlation, causal skeleton, and causality identification. Evaluations of 19 leading LLMs reveal that, while closed-source LLMs show potential for simple causal relationships, they significantly lag behind traditional algorithms on larger-scale networks (

>50

nodes). Specifically, LLMs struggle with collider structures but excel at chain structures, especially at long-chain causality analogous to Chain-of-Thought techniques. This supports the current prompt approaches while suggesting directions to enhance LLMs' causal reasoning capability. Furthermore, CausalBN-Bench incorporates background knowledge and training data into prompts to thoroughly unlock LLMs' text-comprehension ability during evaluation, whose findings indicate that, LLM understand causality through semantic associations with distinct entities, rather than directly from contextual information or numerical distributions.

Abstract

Causal Reasoning in Language Models

Explore similar work

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

Towards a Universal Causal Reasoner