The Illusion of $\textit{What If}$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs

Yucheng Wang; Yuetian Du; Zhengyi Liu; Rongyu Zhang; Bing Zhao; Boyu Yang; Ming Kong; Lin Qu

cs.AIAug 28, 2026

The Illusion of $\textit{What If}$ : Evaluating the Breakdown of Counterfactual Reasoning in LLMs

Authors: Yucheng Wang, Yuetian Du, Zhengyi Liu, Rongyu Zhang, Bing Zhao, Boyu Yang, Ming Kong, Lin Qu, +3 more

Organizations: Zhejiang University · Alibaba Group · City University of Hong Kong

Abstract

Counterfactual reasoning requires models to reason beyond the observed world and explain how altered conditions propagate through downstream consequences. Existing benchmarks largely target bounded settings with fixed variables or single gold outcomes, overlooking open-domain scenarios requiring causal-process evaluation. To this end, we present $\textbf{WhatIfBench}$ , a diagnostic benchmark for open-domain, open-form, long-horizon counterfactual causal reasoning, containing 220 what-if questions across STEM, HSS, and Hybrid scenarios. To evaluate free-form responses, we further propose $\textbf{PRISM}$ , which first converts each natural-language explanation into a Response-Derived Semantic Causal Graph of events, states, and mechanisms. On top of this graph, PRISM then jointly applies a Process Metric assessing graph-level causal validity and a Rubric Metric assessing answer-level explanatory adequacy. Evaluating six frontier LLMs with this framework, we find that WhatIfBench remains far from saturated: even the strongest model reaches only a 64.62% final score. Further analysis reveals persistent causal gaps, premise drift, and topology fragmentation, suggesting that fluent counterfactual narratives often mask fragile causal processes. The benchmark, code, and evaluation scripts are available at WhatIfBench.

Figures & tables

Figure 1: Comparison of what-if reasoning evaluation paradigms. (a) Existing benchmarks focus on outcome-level evaluation for short-horizon reasoning. (b) WhatIfBench shifts the focus to process-level evaluation for long-horizon reasoning by parsing model outputs into causal graphs and checking causal edges.

Figure 2: Overview of WhatIfBench and PRISM. WhatIfBench contains open-form counterfactual queries spanning STEM, HSS, and Hybrid scenarios. For each query, a model generates a free-form response, which PRISM evaluates through two complementary components: the Process Metric (PM), which parses the response into a response-derived semantic causal graph and validates its causal edges, and the Rubric Metric (RM), which scores answer-level explanatory adequacy using query-specific rubrics. The final score combines PM and RM, enabling evaluation of both causal-process validity and answer-level explanatory adequacy.

Target	#	Pass	Revised	Rejected	Valid
Questions	611	14.9	21.1	64.0	36.0
Rubrics	220	92.3	7.7	0.0	100.0

Table 1: Quality control results for WhatIfBench. All values except # are percentages at the audit-target level. Valid denotes Pass + Revised.

Model	STEM		HSS		Hybrid		Overall		Final
Model	PM	RM	PM	RM	PM	RM	PM	RM	Final
GLM-5.1	57.72	52.19	49.63	46.93	52.83	49.92	54.07	49.92	51.99
DeepSeek-V4-Pro	64.43	55.81	49.53	52.74	39.42	54.33	55.78	54.50	55.12
Qwen3-Max	64.92	48.32	49.42	43.17	55.45	48.72	58.07	46.45	52.26
Gemini-3.1-Pro-Preview	68.37	53.75	52.28	50.61	51.24	50.72	60.43	52.24	56.33
Claude-Opus-4.7	69.13	56.04	57.72	52.02	56.08	58.48	63.39	54.82	59.11

Table 2: Main results on WhatIfBench. We report the Process Metric (PM) and Rubric Metric (RM) across STEM, HSS, Hybrid, and overall categories, together with the final aggregated score. All scores are percentages on a 0–100 scale. The best result in each column is shown in bold , and the second-best result is underlined .

Panel of Table 3 — Table 3: Failure mode distribution among weak WhatIfBench responses. Percentages are computed over responses with final PRISM score below 60. Labels are not mutually exclusive.

Figure 3: Topology of response-derived causal graphs across models. We visualize the distributions of two graph-level structural statistics extracted from parsed model responses: (a) the number of nodes, measuring the scale of instantiated causal factors, and (b) the longest directed path length, measuring the depth of causal propagation.

Audit Set	Node	Edge	Graph	Causal
	Validity	Validity	Validity	Verification
Human1	99.6	88.7	98.3	94.2
Human2	100.0	92.3	98.3	93.4
Human3	98.7	89.5	100.0	94.7

Table 4: Human audit of response-to-graph parsing quality. All values are percentages. Node, Edge, Graph, and Causal Verification denote the validity of extracted nodes, directed edges, graph structure, and causal relations, respectively.

Score	Pair	Pearson $r$	Spearman $\rho$	MAE
PM	H–H	0.918	0.910	0.075
PM	P–H	0.812	0.846	0.101
RM	H–H	0.921	0.888	0.029
RM	P–H	0.875	0.726	0.034
Final	H–H	0.937	0.903	0.037
Final	P–H	0.833	0.815	0.059

Table 5: Correlation with human judgments. H–H denotes Human–Human agreement, and P–H denotes PRISM–Human agreement.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Scenario distribution of WhatIfBench. The benchmark contains 220 open-form what-if questions across STEM, HSS, and Hybrid scenarios. STEM accounts for 51.4% of the dataset, followed by HSS at 35.9% and Hybrid at 12.7%.

Figure 5: Fine-grained distribution of HSS questions. HSS scenarios are divided into State, Regime, and Macrohistory; War and Geopolitics; Dynastic Leadership; Domestic Politics and Institutions; and Culture, Technology, and Society. State, Regime, and Macrohistory is the largest HSS subcategory at 40.5%.

Figure 6: Fine-grained distribution of STEM questions. STEM scenarios are divided into Physical and Planetary Systems, Life Sciences and Human Body, Society, Computation, and Logistics, and Engineering, Transport, and Design. Physical and Planetary Systems forms the largest STEM subcategory at 49.6%.

Figure 7: Lexical overview of WhatIfBench questions. The word cloud visualizes frequent terms in normalized what-if questions, offering a qualitative view of the benchmark’s topical coverage across scientific, historical, geopolitical, technological, and cross-domain scenarios.

Benchmark	Open- domain	Open- form	Long- horizon	Process-level Evaluation	Question-specific Rubrics
CRASS ( Frohberg and Binder, 2022 )	$\times$	$\times$	$\times$	$\times$	$\times$
IfQA ( Yu et al., 2023 )	✓	$\times$	$\times$	$\times$	$\times$
CLadder ( Jin et al., 2023 )	$\times$	$\times$	$\times$	$\times$	$\times$
CofCA ( Wu et al., 2025 )	$\times$	$\times$	$\times$	✓	$\times$
CounterBench ( Chen et al., 2025 )	$\times$	$\times$	✓	$\times$	$\times$
WhatIfBench	✓	✓	✓	✓	✓

Appendix

Table 6: Comparison of counterfactual reasoning benchmark characteristics.

Model	WhatIfBench		IfQA		CounterBench
Model	Final $\uparrow$	Rank	EM $\uparrow$	Rank	Acc. $\uparrow$	Rank
GLM-5.1	51.99	6	89.71	6	80.00	3
DeepSeek-V4-Pro	55.12	4	90.43	4	85.71	1
Qwen3-Max	52.26	5	93.00	1	76.74	6
Gemini-3.1-Pro-Preview	56.33	3	90.00	5	78.33	5
Claude-Opus-4.7	59.11	2	91.86	3	79.97	4

Appendix

Table 7: Performance and rankings of frontier LLMs across counterfactual reasoning benchmarks.

Model	Architecture	STEM		HSS		Hybrid		Overall		Final
		PM	RM	PM	RM	PM	RM	PM	RM
Qwen3-4B	Dense	48.87	36.18	45.72	32.88	39.16	36.80	46.59	35.02	40.80
Qwen3-8B	Dense	53.08	34.48	42.49	32.54	55.10	37.20	49.36	34.06	41.71
Gemma-3-12B-it	Dense	49.45	41.73	43.92	41.61	44.72	43.28	46.85	41.86	44.36
GPT-oss-20B	MoE	52.47	43.37	42.76	36.54	46.26	38.64	48.20	40.41	44.26

Appendix

Table 8: Performance of smaller models on WhatIfBench across model architectures.

Method	STEM		HSS		Hybrid		Overall		Final
	PM	RM	PM	RM	PM	RM	PM	RM
Vanilla	72.77	64.00	57.38	61.98	65.32	64.80	65.94	63.31	64.62
Long Output	73.17	64.42	57.13	61.87	67.49	65.60	66.69	63.73	65.21
Short Output	73.31	64.30	58.35	62.46	69.65	65.20	67.47	63.69	65.58
SD	0.28	0.22	0.64	0.31	2.17	0.40	0.77	0.23	0.48

Appendix

Table 9: Effect of response length on PRISM scores. SD is computed across the three response-length settings.

Method	STEM		HSS		Hybrid		Overall		Final
	PM	RM	PM	RM	PM	RM	PM	RM
Paraphrase	2.65	10.00	1.83	10.10	0.00	10.00	2.05	10.04	6.04

Appendix

Table 10: PRISM stress test with paraphrase-only responses.

Method	STEM		HSS		Hybrid		Overall		Final
	PM	RM	PM	RM	PM	RM	PM	RM
Vanilla	72.77	64.00	57.38	61.98	65.32	64.80	65.94	63.31	64.62
CausalCoT	63.98	68.34	68.32	67.88	68.78	70.00	66.12	68.37	67.25
CoIn	69.16	62.63	69.60	61.15	73.65	64.84	69.61	62.33	65.97

Appendix

Table 11: Effect of counterfactual-specific prompting strategies on GPT-5.5.

Model	STEM		HSS		Hybrid		Overall		Final	Rank
	PM	RM	PM	RM	PM	RM	PM	RM
GLM-5.1	59.59	52.19	48.64	47.00	53.95	50.32	54.81	49.99	52.40	6
DeepSeek-V4-Pro	65.75	55.50	50.91	52.94	42.05	54.75	57.51	54.42	55.97	4
Qwen3-Max	67.75	48.27	53.99	43.00	56.85	48.56	61.33	46.34	53.83	5
Gemini-3.1-Pro-Preview	67.89	54.18	51.57	50.32	50.89	50.24	59.83	52.29	56.06	3
Claude-Opus-4.7	68.88	56.11	56.89	52.37	55.57	58.88	62.87	54.94	58.90	2

Appendix

Table 12: PRISM scores and model rankings using Claude-Sonnet-5 as the evaluator.

Panel of Figure 8 — Figure 8: Model response generation prompt used for WhatIfBench evaluation.

Panel of Figure 9 — Figure 9: Response-to-Graph Parser Prompt used to construct response-derived semantic causal graphs.

Panel of Figure 10 — Figure 10: Unified judge prompt template used for Process Metric (PM) edge evaluation within the PRISM framework.

Panel of Figure 11 — Figure 11: Unified judge prompt template used for Rubric Metric (RM) evaluation within the PRISM framework.

Panel of Figure 12 — Figure 12: Example WhatIfBench question and its query-dependent rubric.

Panel of Figure 13 — Figure 13: Model response generated for the selected WhatIfBench case.

Panel of Figure 14 — Figure 14: Abstracted response-derived semantic causal graph for the selected case. Nodes represent interpretable causal modules, while arrows show the main causal and discourse relations expressed in the response.

Panel of Figure 15 — Figure 15: Rubric-based judgment for the selected model response under the RM component of PRISM.

Explore similar work

Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency
Feb 18, 2026Victoria Lin, Xinnuo Xu, Rachel Lawrence +4Counterfactual Reasoning in Language Models Causal Reasoning in Language Models
Carnegie Mellon University · Microsoft Research Cambridge · Microsoft Research India
CoBe: A Benchmark For Conversational Counterfactual Text Editing
Oct 9, 2026Arvind Raghavan, Inwoo Hwang, Shreyas Havaldar +6Causal Counterfactual Generation Counterfactual Reasoning in Language Models
Causal Artificial Intelligence Lab Columbia University
Synergizing Large Language Models and Theory for Human-like Causal Reasoning
May 13, 2025Yanxi Zhang, Xin Cong, Zhong Zhang +3Causal Reasoning in Language Models Causal Reasoning
Center for Data Science, AAIS, Peking University · Wangxuan Institute of Computer Technology, Peking University · Tsinghua University +1

Feb 18, 2026J/K move · Enter open · S save

Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency

Victoria Lin, Xinnuo Xu, Rachel Lawrence, Risa Ueno+3

Carnegie Mellon University · Microsoft Research Cambridge · Microsoft Research India

Counterfactual Reasoning in Language Models Causal Reasoning in Language Models Reasoning Consistency in Language Models Causal Interventions in Language Models

Open paper AskPDF arXiv

Despite their strong performance on reasoning benchmarks, large language models (LLMs) have proven brittle when presented with counterfactual questions, suggesting weaknesses in their causal reasoning ability. While recent work has demonstrated that labeled counterfactual tasks can be useful benchmarks of LLMs' causal reasoning, producing such data at the scale required to cover the vast potential space of counterfactuals is limited. In this work, we introduce double counterfactual consistency (DCC), a lightweight inference-time method for measuring and guiding the ability of LLMs to reason causally. Without requiring labeled counterfactual data, DCC verifies a model's ability to execute two important elements of causal reasoning: causal intervention and counterfactual prediction. Using DCC, we evaluate the causal reasoning abilities of various leading LLMs across a range of reasoning tasks and interventions. Moreover, we demonstrate the effectiveness of DCC as a training-free test-time rejection sampling criterion and show that it can directly improve performance on reasoning tasks across multiple model families.

Abstract

Explore similar work

Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency

CoBe: A Benchmark For Conversational Counterfactual Text Editing

Synergizing Large Language Models and Theory for Human-like Causal Reasoning