Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification, formulating a valid research design under stated assumptions, and estimation, implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 174 queries across 139 real-world datasets, curated from 73 peer-reviewed research papers and three widely used causal-inference textbooks. For each query, a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art LLM show that, while the model correctly identifies the high-level strategy in 81% of cases, full identification-specification correctness drops to only 38%, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.
Figures & tables
Benchmark
# Queries
Real data
ID eval
Quant. eval
Design- specific
Designs covered
QRData [ 64 ]
411
✓
Partial
Mixed
CausalBench [ 89 ]
495
Partial
Mixed
CLadder [ 51 ]
10k
✓
Graph-based
Corr2Cause [ 52 ]
413k
✓
Graph-based
CausalGraph2LLM [ 80 ]
700k+
✓
Graph-based
CausalReasoningBenchmark
174
✓
✓
✓
✓
IV, RDD, DiD, CE, RCT
Table 1: Comparison of CausalReasoningBenchmark with related benchmarks. “ID eval” indicates whether identification is evaluated separately from estimation. “Real data” indicates whether the benchmark uses real-world (non-synthetic) datasets. “Quant. eval” means evaluating causal effect estimation from data (e.g., ATE / ATT / LATE / CATE). “Design-specific” indicates whether the benchmark requires specification of design-specific elements (e.g., instruments, running variables).
Source group
#queries
#datasets
Research papers
120
85
Textbook
54
54
Total
174
139
Table 2: CausalReasoningBenchmark queries and datasets by source group.
Identification strategy
#queries
#datasets
Difference-in-Differences
67
37
Regression Discontinuity
44
39
Instrumental Variable
22
22
Conditional Exogeneity
40
40
RCT
1
1
Table 3: CausalReasoningBenchmark composition by identification strategy.
Source group
DiD
RDD
IV
Cond. Exog.
RCT
Research papers
62
39
19
0
0
Textbook
5
5
3
40
1
Table 4: Query counts by source group and identification strategy.
Metric
Value
Identification Metrics
Strategy correct
81.0% (141/174)
Causal quantity correct
74.7% (130/174)
Treatments correct
87.4% (152/174)
Outcomes correct
96.0% (167/174)
Minimal controlling set included
75.9% (132/174)
Table 5: Aggregate evaluation of the GPT-5.3 baseline on all 174 queries. Identification metrics are exact-match or set-based checks against the gold specification; estimation metrics compare the returned effect and uncertainty to the gold solution. Values in brackets denote the interquartile range.
Design
#Queries
Strategy (%)
Full ID (%)
Med. %Err
Difference-in-Differences
67
94.0
59.7
9.7
Regression Discontinuity
44
95.5
11.4
21.0
Instrumental Variable
22
86.4
36.4
50.1
Conditional Exogeneity
40
40.0
30.0
4.0
RCT
1
100.0
100.0
0.0
Overall
174
81.0
37.9
13.5
Table 6: Per-strategy breakdown of the GPT-5.3 baseline. “Strategy” = fraction with correct strategy label; “Full ID” = fraction with fully correct identification specification; “Med. %Err” = median percentage error on the effect estimate.
Error Family
Error Type
Count
Control variables
Missing required controls
42
Included bad / post-treatment controls
10
Estimand
CATE → LATE
28
CLATE → LATE
7
Other estimand errors
9
Strategy
CE → RCT
24
Table 7: Taxonomy of identification errors across all 108 incorrect cases. A single case may contribute to multiple categories. Counts are grouped by error family.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Paper
Design
Title
#queries
[ 59 ]
DiD
Corporate Board Quotas and Gender Equality Policies in the Workplace
2
[ 75 ]
DiD
Deadly Populism: How Local Political Outsiders Drive Duterte’s War on Drugs in the Philippines
2
[ 23 ]
DiD
Does Compliance Pay? Social Standards and Firm-Level Trade
2
[ 40 ]
DiD
Does Direct Democracy Hurt Immigrant Minorities? Evidence from Naturalization Decisions in Switzerland
2
[ 71 ]
DiD
Education or Indoctrination? The Violent Origins of Public School Systems in an Era of State-Building
2
[ 88 ]
DiD
Elite Cleavage and the Rise of Capitalism under Authoritarianism: A Tale of Two Provinces in China
2
Appendix
Table 8: Research papers included in the benchmark. Each row summarizes queries from one paper and design; papers used under multiple designs appear in separate rows.