EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models

Donggyu Lee; Hyeok Yun; Meeyoung Cha; Sungwon Park; Sangyoon Park; Jihee Kim

cs.CLOct 8, 2025

EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models

Authors: Donggyu Lee, Hyeok Yun, Meeyoung Cha, Sungwon Park, Sangyoon Park, Jihee Kim

Organizations: Graduate School of Data Science, KAIST · College of Business, KAIST · Data Science for Humanity Group, MPI-SP · School of Computing, KAIST ·

Abstract

Socio-economic causal effects depend heavily on their institutional and environmental contexts. The same intervention can produce different, even opposite, effects across regulatory regimes, market conditions, time periods, or populations. This poses a challenge for large language models (LLMs) in decision-support roles: can they infer the direction of a causal effect under a specified context, and revise that judgment when the context changes? To address this, we introduce EconCausal, a large-scale benchmark of 10,490 context-annotated causal triplets extracted from 2,595 high-quality empirical studies in top-tier economics and finance journals, constructed through a rigorous four-stage pipeline with multi-run consensus, context refinement, and multi-critic filtering. Across 19 models, we observe source-sign over-transfer in cross-context prediction. While top models reach ~84% accuracy in single-context prediction, mean accuracy is 74.75% overall but drops to 44.27% when source and target signs disagree. On these cases, accuracy is 11.54 percentage points (pp) lower with the signed source example than with the target input alone. Mean accuracy is below 60% under corrupted source signs. Models also struggle to predict study-reported nonsignificant effects (19.16% accuracy). The dataset and benchmark are publicly available at https://anonymous.4open.science/r/econcausal-benchmark-6F12.

Figures & tables

Figure 1: Context-dependent causality in socio-economic settings. Causal effects can vary across high-dimensional and partially observed contexts, posing a challenge for LLM-based reasoning in socio-economic settings.

Figure 2: Overview of the EconCausal dataset extraction pipeline.

Task #	Question	Evaluation Goal
Task 1 Causal Sign Identification	Given context $c$ and a treatment–outcome pair $(T,O)$ , what is the most likely effect category of $T\rightarrow O$ ?	Can the model predict the study-reported effect category under its documented context? ( RQ1 )
Example Given context $c$ : “minimum-wage increase ( $4.25 to$ 5.05) on fast-food establishments in New Jersey in April 1992, …, amid a concurrent regional recession”, $T$ : minimum-wage increase, $O$ : employment, what is the causal sign of $T$ to $O$ ?
Task 2 Context-Dependent Sign Prediction	Given a signed finding under context $c_{1}$ , predict the effect category for the same or a comparable relation under context $c_{2}$ .	Can the model distinguish target-study findings from related source findings when their reported signs differ? ( RQ2 )
Example Given the triplet ( $T$ : higher subsidy, $O$ : insurance take-up rate, $+$ ) under context $c_{1}$ : “a two-year (2010–2011) randomized field experiment in Jiangxi province, China, …”, what is the sign of the pair ( $T$ , $O$ ) under a different context $c_{2}$ : “Massachusetts’ subsidized insurance exchange CommCare from 2009–2013, …”?
Task 3 Misinformation-Robust Sign Prediction	Given the triplet ( $T$ , $O$ , sign) under context $c_{1}$ with its sign deliberately corrupted (noise), what is the causal sign of the target relation under a different context $c_{2}$ ?	Do target-study predictions remain accurate when the source sign is corrupted? ( RQ3 )
Example Same as in Task #2 example, except the given triplet is ( $T$ : higher subsidy, $O$ : insurance take-up rate, $-$ ), where the sign is incorrect.

Table 1: Overview of the three EconCausal benchmark tasks. A complete instantiated example, including input contexts and model responses, is provided in Appendix K.3 .

Closed-Source LLMs
			Task 2
Model	Task 1 (Econ)	Task 1 (Finance)	Overall	Sign-Mismatch	Task 3
Gemini 3.1 Pro	0.838 (0.609)	0.831 (0.606)	0.906 (0.840)	0.806 (0.805)	0.877 (0.733)
Gemini 3 Flash	0.839 (0.592)	0.821 (0.592)	0.863 (0.674)	0.621 (0.566)	0.736 (0.539)
Gemini 2.5 Pro	0.774 (0.500)	0.757 (0.480)	0.776 (0.547)	0.581 (0.501)	0.685 (0.455)
Gemini 2.5 Flash	0.752 (0.460)	0.748 (0.475)	0.764 (0.494)	0.476 (0.384)	0.453 (0.315)

Table 2: Main results on EconCausal benchmark. We report accuracy (Macro F1) for each task. “Sign-Mismatch” denotes the subset where the example sign contradicts the ground-truth sign of the target question. Bold indicates the best in each column.

	Correct	Example sign error	Other error
Target only (Task 1 setting, $N=415$ )	75.10	8.41	16.49
With example (Task 2, $N=415$ )	74.75	13.16	12.09
$\Delta$ (pp)	$-0.36$	$+4.76$	$-4.40$
Sign-matched ( $N=291$ )	87.74	0.00	12.26
Sign-mismatched ( $N=124$ )	44.27	44.06	11.67

Table 3: Prediction shares (%) across 19 models, overall (top) and by source–target sign agreement (bottom).

	Target only	Unrelated source	Related source
Sign-matched ( $N=291$ )	83.32	79.42	87.74
Sign-mismatched ( $N=124$ )	55.81	51.83	44.27

Table 4: Target-sign accuracy (%) across 19 models. Sign agreement is defined using the original related source.

	Error type	Task 2	Task 3
A	Input/variable misinterpretation	9.6	5.9
B	Unqualified source transfer	26.1	10.4
C	Unsupported study-specific claims	17.5	15.2
D	Invalid economic/logical inference	7.9	4.9
E	Effect-label/output inconsistency	9.6	10.4

Table 5: Non-exclusive error frequencies (%) among incorrect predictions.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Economics journals
Journal	# Papers
American Economic Review	751
Quarterly Journal of Economics	440
Journal of Political Economy	258
Review of Economic Studies	177
Econometrica	123

Appendix

Table 6: Distribution of source papers by journal and domain.

Score-level agreement
Metric	Value
Per-criterion MAE	0.229
Sum-of-scores MAE	1.018
Filtering and approval outcomes
Economist approval rate before filtering	83.5%
Step 4 acceptance rate	76.2%

Appendix

Table 7: Economist validation summary for LLM-critic scores and Step 4 filtering.

Criterion	Mean score	Share $\geq 2$
Variable extraction	2.87 / 3	97.57%
Direction	2.99 / 3	99.51%
Sign score	2.93 / 3	97.57%
Causality	2.83 / 3	94.66%
Main claim	2.72 / 3	90.78%
Context	2.94 / 3	98.06%

Appendix

Table 8: Criterion-level economist validation scores. Mean scores are reported on a 0–3 scale, and the final column reports the share of triplets receiving a score of at least 2.

Criterion	Agreement with LLM
Treatment containment (Q1)	6/10
Treatment comparability (Q2)	9/10
Outcome containment (Q3)	9/10
Outcome comparability (Q4)	10/10
Relation comparability (Q5)	8/10
Overall Agreement	8/10

Appendix

Table 9: Agreement between the economist majority and the LLM across ten candidate pairs.

Cutoff ( $N$ )	Group	T2 Overall	T2 Mismatch	T3
$\geq 6$ (415)	Closed	0.779 (0.558)	0.507 (0.444)	0.637 (0.454)
	Open	0.694 (0.397)	0.332 (0.218)	0.501 (0.279)
$\geq 7$ (170)	Closed	0.759 (0.556)	0.464 (0.412)	0.588 (0.431)
	Open	0.696 (0.437)	0.302 (0.222)	0.480 (0.281)
$\geq 8$ (17)	Closed	0.868 (0.588)	0.417 (0.201)	0.556 (0.393)
	Open	0.815 (0.505)	0.286 (0.095)	0.426 (0.245)

Appendix

Table 10: Sensitivity to pair-comparability thresholds based on the sum of Q2, Q4, and Q5 scores. Values report accuracy on the first line and macro-F1 in parentheses on the second line, averaged across closed-source (12) and open-source (7) models.

N

denotes the number of retained source–target pairs.

	Task 1		Task 2		Task 3
	Econ	Finance	Overall	Sign mismatch
Economists	35.6	70.0	70.0	47.2	23.3
Closed LLMs	55.8	91.7	68.9	36.3	50.3
Open LLMs	47.9	85.5	61.0	22.6	39.0

Appendix

Table 11: Accuracy (%) of economists and LLMs on the economist-evaluation questions. Three economists each answered ten questions per task. LLM results are averaged across models within each group.

Task	Shared accuracy	Individualized accuracy	Agreement (%)
Task 1	7/15 (46.7)	8/15 (53.3)	73.3
Task 2	10/15 (66.7)	11/15 (73.3)	86.7
Task 3	3/15 (20.0)	4/15 (26.7)	33.3

Appendix

Table 12: Economist results by question allocation. Accuracy entries show correct/total responses (percentage), pooled across the three economists. Each economist answered five shared and five individualized questions per task. Task 1 combines Economics and Finance. Agreement is reported only for the five shared questions answered by all three economists.

Panel of Figure 3 — Figure 3: GPT-4o uncertainty analysis on Task 1.

Model	Main	Post-cutoff	$\Delta$
Gemini 3.1 Pro	90.60	82.18	-8.42
Gemini 3 Flash	86.27	79.21	-7.06
Gemini 2.5 Pro	77.59	72.28	-5.31
Gemini 2.5 Flash	76.39	80.20	+3.81
GPT-5.2	76.87	69.31	-7.56
GPT-5 mini	75.66	78.22	+2.56

Appendix

Table 13: Task 2 accuracy (%) on the main 415-pair and post-cutoff 101-pair sets.

\Delta

denotes post-cutoff minus main accuracy in percentage points.

Model	Title	Author	Journal	Joint	$r$
GPT-4o mini	8.8	7.4	19.1	1.5	0.105
GPT-4o	8.8	29.4	17.6	5.9	0.008
GPT-5 nano	5.9	10.3	5.9	1.5	0.087
GPT-5 mini	19.1	25.0	22.1	14.7	0.144
GPT-5.2	11.8	25.0	19.1	7.4	-0.098
Gemini 3.1 Pro	82.4	86.8	77.9	77.9	0.012

Appendix

Table 14: Source-paper recall on 68 Task 1 items. Recall rates are percentages.

r

denotes the within-model correlation between joint recall and Task 1 correctness; undefined values are shown as –.

Statistic	Estimate
Coefficient ( $\beta$ )	$0.000256$
Clustered SE	$0.000357$
95% CI	$[-0.000443,\;0.000955]$
$p$ -value	$0.4724$

Appendix

Table 15: Logistic regression of Task 1 correctness on raw source-paper citation counts, controlling for year, model, gold-sign, and journal fixed effects. Standard errors are clustered at the paper level.

		Task 2		Task 3
Group	Prompt	Acc.	F1	Acc.	F1
Closed (12)	BASE	77.85	55.80	63.69	45.42
	SIMPLE	80.54	53.39	68.17	44.99
	EXPERT	78.55	56.55	66.73	47.46
Open (7)	BASE	69.43	39.67	50.05	27.91
	SIMPLE	71.12	38.36	57.68	29.31

Appendix

Table 16: Prompt sensitivity on Tasks 2 and 3. Accuracy and macro-F1 (%) are equally weighted model averages.

Closed models
Model	100 words	300 words	$\Delta$
Gemini 3.1 Pro Preview	90.60	91.08	+0.48
Gemini 3 Flash Preview	86.27	88.43	+2.17
Gemini 2.5 Pro	77.59	79.52	+1.93
Gemini 2.5 Flash	76.39	76.63	+0.24
GPT-5.2	76.87	79.76	+2.89

Appendix

Table 17: Task 2 accuracy (%) under different context-length caps on the same 415 pairs.

\Delta

is in percentage points.

Model / Input	Economics	Finance
LLM mean (19)	0.690 (0.428)	0.675 (0.421)
Closed LLMs (12)	0.734 (0.485)	0.729 (0.478)
Open LLMs (7)	0.613 (0.330)	0.582 (0.322)
TF-IDF: Context	0.460 (0.284)	0.473 (0.296)
TF-IDF: T/O	0.436 (0.275)	0.405 (0.263)
TF-IDF: Context + T/O	0.465 (0.281)	0.467 (0.282)

Appendix

Table 18: Task 1 accuracy (macro-F1), on a 0–1 scale.

	Changed (%)	Observed agree. (%)	Permuted agree. (%)	Excess (pp)
Closed (12)	41.77	29.17	16.02	+13.15
Open (7)	60.91	37.63	18.71	+18.92
Overall (19)	48.82	32.29	17.01	+15.28

Appendix

Table 19: Task 3 within-instance source-sign sensitivity. Changed denotes at least one prediction change across the three variants.

Model	T1 Econ	T1 Finance	T2 Overall	T2 Mismatch	T3
Gemini 3.1 Pro	0.838 [0.813,0.863]	0.831 [0.803,0.860]	0.906 [0.866,0.939]	0.806 [0.725,0.881]	0.877 [0.827,0.917]
Gemini 3 Flash	0.839 [0.815,0.864]	0.821 [0.791,0.850]	0.863 [0.813,0.905]	0.621 [0.525,0.725]	0.736 [0.678,0.787]
Gemini 2.5 Pro	0.774 [0.744,0.803]	0.757 [0.724,0.790]	0.776 [0.721,0.827]	0.581 [0.488,0.678]	0.685 [0.620,0.742]
Gemini 2.5 Flash	0.752 [0.721,0.782]	0.748 [0.717,0.778]	0.764 [0.698,0.824]	0.476 [0.381,0.583]	0.453 [0.402,0.501]
GPT-5.2	0.729 [0.697,0.761]	0.734 [0.702,0.764]	0.769 [0.702,0.830]	0.476 [0.384,0.573]	0.664 [0.586,0.736]
GPT-5 mini	0.699 [0.666,0.732]	0.701 [0.667,0.735]	0.757 [0.692,0.815]	0.427 [0.331,0.540]	0.598 [0.539,0.657]

Appendix

Table 20: Accuracy: point estimates with their 95% confidence intervals in brackets.

Task	$+$	$-$	None	Mixed	Overall
Task 1 (Econ)	80.26	66.18	18.09	13.68	68.96
Task 1 (Finance)	80.63	64.53	10.02	24.85	67.48
Task 2	83.49	72.57	24.36	49.12	74.75
Task 3	65.93	52.79	24.19	53.22	58.66

Appendix

Table 22: Per-class recall (%) and overall accuracy, averaged across 19 models.

Figure 4: Distribution of ground-truth causal signs in EconCausal benchmark (Task 1/2/3). Each triplet is labeled as positive (

+

), negative (

-

), none, or Mixed based on the authors’ preferred empirical specification.

JEL	Field	Accuracy	N
B	History of Economic Thought	86.8%	2
N	Economic History	77.6%	63
K	Law and Economics	74.0%	89
R	Urban/Rural/Regional	75.4%	72
F	International Economics	72.9%	216
L	Industrial Organization	70.6%	155

Appendix

Table 23: Accuracy by JEL code.

Causal Reasoning in Language Models

Causal reasoning in language models studies how models identify cause-and-effect structure, predict interventions, and answer counterfactual questions.

See all 102 papers

Ideological Bias in LLMs' Economic Causal Reasoning
Apr 23, 2026Donggyu Lee, Hyeok Yun, Jungwon Kim +4Political Bias in Language Models Language Model Bias Evaluation
Graduate School of Data Science, KAIST, Daejeon, South Korea · College of Business, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea +1
CausalVerify: End-to-End Verification of Causal Analyses by Language Models
Sep 7, 2026Yonghong Zhang, Ricardo Correia, Isabel M. Parra +1LLM Evaluation Causal Effect Estimation
Department of Finance Universidad Autónoma de Madrid Madrid, Spain · Spanish National Research Council (CSIC) Madrid, Spain
METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models
Apr 13, 2026Pengfeng Li, Chen Huang, Chaoqun Hao +4Causal Reasoning in Language Models LLM Evaluation
College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore

Apr 23, 2026J/K move · Enter open · S save

Ideological Bias in LLMs' Economic Causal Reasoning

Donggyu Lee, Hyeok Yun, Jungwon Kim, Junsik Min+3

Graduate School of Data Science, KAIST, Daejeon, South Korea · College of Business, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea+1

Political Bias in Language Models Language Model Bias Evaluation Causal Reasoning Causal Inference

Open paper AskPDF arXiv

Do large language models (LLMs) exhibit systematic ideological bias when reasoning about economic causal effects? As LLMs are increasingly used in policy analysis and economic reporting, where directionally correct causal judgments are essential, this question has direct practical stakes. We present a systematic evaluation by extending the EconCausal benchmark with ideology-contested cases - instances where intervention-oriented (pro-government) and market-oriented (pro-market) perspectives predict divergent causal signs. From 10,490 causal triplets (treatment-outcome pairs with empirically verified effect directions) derived from top-tier economics and finance journals, we identify 1,056 ideology-contested instances and evaluate 20 state-of-the-art LLMs on their ability to predict empirically supported causal directions. We find that ideology-contested items are consistently harder than non-contested ones, and that across 17 of 20 models, accuracy is systematically higher when the empirically verified causal sign aligns with intervention-oriented expectations than with market-oriented ones. Moreover, when models err, their incorrect predictions disproportionately lean intervention-oriented, and this directional skew is not eliminated by one-shot in-context prompting. These results highlight that LLMs are not only less accurate on ideologically contested economic questions, but systematically less reliable in one ideological direction than the other, underscoring the need for direction-aware evaluation in high-stakes economic and policy settings.

Abstract

Causal Reasoning in Language Models

Explore similar work

Ideological Bias in LLMs' Economic Causal Reasoning

CausalVerify: End-to-End Verification of Causal Analyses by Language Models

METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models