Thinking Fast, Thinking Wrong: Intuitiveness Modulates LLM Counterfactual Reasoning in Policy Evaluation

Yanjie He

cs.AIApr 12, 2026

Thinking Fast, Thinking Wrong: Intuitiveness Modulates LLM Counterfactual Reasoning in Policy Evaluation

Authors: Yanjie He

Organizations: Independent Researcher

Abstract

Large language models (LLMs) are increasingly used for causal and counterfactual reasoning, yet their reliability in real-world policy evaluation remains underexplored. We construct a benchmark of 40 empirical policy evaluation cases drawn from economics and social science, each grounded in peer-reviewed evidence and classified by intuitiveness -- whether the empirical finding aligns with (obvious), is unclear relative to (ambiguous), or contradicts (counter-intuitive) common prior expectations. We evaluate four frontier LLMs across five prompting strategies with 8,000 experimental trials and analyze the results using mixed-effects logistic regression. Our findings reveal three key results: (1) a chain-of-thought (CoT) paradox, where chain-of-thought prompting dramatically improves performance on obvious cases but this benefit is substantially attenuated on counter-intuitive ones (interaction OR = 0.278, $p < 0.001$ ); (2) intuitiveness as the dominant factor, with case-level variance exceeding that of model choice or prompting strategy (ICC = 0.671); and (3) a knowledge-reasoning dissociation, where citation-based familiarity is unrelated to accuracy ( $p = 0.84$ ), suggesting models possess relevant knowledge but fail to reason with it when findings contradict intuition. We frame these results through the lens of dual-process theory (System 1 vs. System 2) and argue that current LLMs' "slow thinking" achieves only partial inhibition of intuitive priors -- producing the form of deliberative reasoning without fully delivering its substance.

Figures & tables

Intuitiveness	One-sided	Two-sided
Obvious	0	13
Ambiguous	1	5
Counter-intuitive	13	8
Total	14	26

Table 1: Distribution of 40 benchmark cases by intuitiveness and test type. One-sided cases ask a YES/NO question; two-sided cases ask for directional prediction (INCREASE / DECREASE / NO SIGNIFICANT CHANGE).

Model	Overall	Two-sided	TS c-intuit.
GPT-5.2	96.3%	94.3%	94.5%
Claude Sonnet 4.5	79.8%	69.6%	73.5%
Claude Opus 4.6	74.7%	61.7%	64.8%
GPT-4.1	67.6%	51.7%	43.2%

Table 2: Accuracy by model. TS c-intuit. = two-sided counter-intuitive cases only. GPT-5.2 maintains high performance even on counter-intuitive cases. Claude Sonnet 4.5 outperforms Claude Opus 4.6 on counter-intuitive cases despite being a smaller model, suggesting model scale alone does not determine susceptibility to intuitive priors.

Prompt	Overall	Two-sided	TS c-intuit.
P4 Structured	96.1%	94%	85%
P3 CoT	95.5%	93.1%	83.8%
P1 Naïve	80.8%	72%	70%
P2 Expert	68.6%	53.3%	64.7%
P5 Adversarial	57.1%	34.2%	41.6%

Table 3: Accuracy by prompt strategy. The expert persona (P2) underperforms the naïve baseline (P1), while the adversarial prompt (P5) severely degrades overall accuracy. CoT and structured prompts remain effective on counter-intuitive cases, though with attenuated benefit (§ 5.4 ).

Interaction	OR	95% CI	$p$
P3 CoT $\times$ c-intuit.	0.278	[0.140, 0.560]	$<$ 0.001
P4 Struct. $\times$ c-intuit.	0.244	[0.110, 0.520]	$<$ 0.001
P5 Advers. $\times$ c-intuit.	2.83	[1.70, 4.71]	$<$ 0.001
Random effects
$\sigma^{2}_{u}$ (case intercept)	6.706
ICC	0.671

Table 4: Key interaction terms from the mixed-effects logistic regression (Eq. 1 ). CoT and structured prompting show a substantial reduction of benefit on counter-intuitive cases (OR

<

1). The P5 positive interaction (

p

<

0.001) is confounded by test-type imbalance (see text).

Figure 1: The CoT paradox: change in accuracy from naïve to CoT prompting by intuitiveness category (two-sided cases only). CoT improves accuracy by +26.7 percentage points on obvious cases but only +13.8 points on counter-intuitive cases, with this gap driven by CoT partially reinforcing rather than fully overriding intuitive priors.

Figure 2: Heatmap of accuracy by model and case across all prompt strategies and repetitions. Red cells indicate systematic failures; cases are sorted by intuitiveness. Notable failure clusters appear on Cases 23 (daycare fines) and 36 (right-to-carry laws).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

#	Policy	Country	Domain	Type	Intuit.	GT
1	NJ Minimum Wage Increase (1992)	US	Labor	1-sided	CI	NO
2	Oregon Medicaid Expansion (2008)	US	Health	1-sided	CI	NO
3	EITC Expansion (1986)	US	Labor	2-sided	Obvious	DECREASE
4	TN STAR Class Size (1985)	US	Education	2-sided	Obvious	DECREASE
5	Head Start Preschool (1965)	US	Education	2-sided	Obvious	INCREASE
6	Mexico PROGRESA (1997)	Mexico	Education	2-sided	Obvious	DECREASE

Appendix

Table 5: Complete list of 40 benchmark cases. Type: 1-sided = YES/NO, 2-sided = directional. Intuit.: CI = counter-intuitive, Ambig. = ambiguous. GT = ground truth answer.

Figure 3: Accuracy by model and intuitiveness category on two-sided cases. GPT-5.2 maintains high accuracy across categories, while GPT-4.1 drops to 43.2% on counter-intuitive cases.

Figure 4: Accuracy by prompt strategy and intuitiveness category on two-sided cases. CoT and structured prompts show uniformly high accuracy on obvious cases, but their advantage diminishes on counter-intuitive cases.

Figure 5: Overall accuracy by model across all conditions. Error bars show 95% binomial confidence intervals.

Figure 6: Overall accuracy by prompt strategy across all conditions. Structured reasoning (P4) and CoT (P3) substantially outperform naïve (P1), expert (P2), and adversarial (P5) prompts.

ID	Background	Familiarity	Accuracy
A1	Economics	5/5	26/40 (65%)
A2	Statistics	3/5	25/40 (63%)
A3	Medicine	2/5	25/40 (63%)
Fleiss’ $\kappa$		0.32 (fair)

Appendix

Table 6: Human annotator characteristics. Familiarity = self-rated familiarity with economics/social science (1–5). The moderate

\kappa

reflects genuine disagreement on counter-intuitive cases, consistent with our task design.

Counterfactual Reasoning in Language Models

Counterfactual reasoning in language models studies how models predict and explain outcomes when facts, evidence, actions, or contexts are changed.

See all 44 papers

Feb 18, 2026

Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency

Despite their strong performance on reasoning benchmarks, large language models (LLMs) have proven brittle when presented with counterfactual questions, suggesting weaknesses in their causal reasoning ability. While recent work has demonstrated that labeled counterfactual tasks can be useful benchmarks of LLMs' causal reasoning, producing such data at the scale required to cover the vast potential space of counterfactuals is limited. In this work, we introduce double counterfactual consistency (DCC), a lightweight inference-time method for measuring and guiding the ability of LLMs to reason causally. Without requiring labeled counterfactual data, DCC verifies a model's ability to execute two important elements of causal reasoning: causal intervention and counterfactual prediction. Using DCC, we evaluate the causal reasoning abilities of various leading LLMs across a range of reasoning tasks and interventions. Moreover, we demonstrate the effectiveness of DCC as a training-free test-time rejection sampling criterion and show that it can directly improve performance on reasoning tasks across multiple model families.

Victoria Lin, Xinnuo Xu, Rachel Lawrence +4

Carnegie Mellon University · Microsoft Research Cambridge · Microsoft Research India

Counterfactual Reasoning in Language Models Causal Reasoning in Language Models Reasoning Consistency in Language Models Causal Interventions in Language Models

Ask PDF

Apr 23, 2026

Ideological Bias in LLMs' Economic Causal Reasoning

Do large language models (LLMs) exhibit systematic ideological bias when reasoning about economic causal effects? As LLMs are increasingly used in policy analysis and economic reporting, where directionally correct causal judgments are essential, this question has direct practical stakes. We present a systematic evaluation by extending the EconCausal benchmark with ideology-contested cases - instances where intervention-oriented (pro-government) and market-oriented (pro-market) perspectives predict divergent causal signs. From 10,490 causal triplets (treatment-outcome pairs with empirically verified effect directions) derived from top-tier economics and finance journals, we identify 1,056 ideology-contested instances and evaluate 20 state-of-the-art LLMs on their ability to predict empirically supported causal directions. We find that ideology-contested items are consistently harder than non-contested ones, and that across 17 of 20 models, accuracy is systematically higher when the empirically verified causal sign aligns with intervention-oriented expectations than with market-oriented ones. Moreover, when models err, their incorrect predictions disproportionately lean intervention-oriented, and this directional skew is not eliminated by one-shot in-context prompting. These results highlight that LLMs are not only less accurate on ideologically contested economic questions, but systematically less reliable in one ideological direction than the other, underscoring the need for direction-aware evaluation in high-stakes economic and policy settings.

Donggyu Lee, Hyeok Yun, Jungwon Kim +4

Graduate School of Data Science, KAIST, Daejeon, South Korea · College of Business, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea +1

Political Bias in Language Models Language Model Bias Evaluation Causal Reasoning Causal Inference

Ask PDF

Mar 16, 2026

InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems

Causal inference in social science relies on end-to-end, intervention-centered research-design reasoning grounded in real-world policy interventions, but current benchmarks fail to evaluate this capability of large language models (LLMs). We present InterveneBench, a benchmark designed to assess such reasoning in realistic social settings. Each instance in InterveneBench is derived from an empirical social science study and requires models to reason about policy interventions and identification assumptions without access to predefined causal graphs or structural equations. InterveneBench comprises 744 peer-reviewed studies across diverse policy domains. Experimental results show that state-of-the-art LLMs struggle under this setting. To address this limitation, we further propose a multi-agent framework, STRIDES. It achieves significant performance improvements over state-of-the-art reasoning models. Our code and data are available at https://github.com/Sii-yuning/STRIDES.

Shaojie Shi, Zhengyu Shi, Lingran Zheng +15

Fudan University · Shanghai Innovation Institute

Causal Reasoning in Language Models Causal Inference Computational Social Science

Ask PDF

Abstract

Counterfactual Reasoning in Language Models

Explore similar work

Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency

Ideological Bias in LLMs' Economic Causal Reasoning

InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems