InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems

InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems

Abstract

Figures & tables

Causal Reasoning in Language Models

Why LLMs Fail at Causal Discovery and How Interventional Agents Escape

Abstract

Causal Reasoning in Language Models

Explore similar work

Appendix

Why LLMs Fail at Causal Discovery and How Interventional Agents Escape

CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Shaojie Shi; Zhengyu Shi; Lingran Zheng; Xinyu Su; Anna Xie; Bohao Lv; Rui Xu; Zijian Chen

cs.CYMar 16, 2026

Authors: Shaojie Shi, Zhengyu Shi, Lingran Zheng, Xinyu Su, Anna Xie, Bohao Lv, Rui Xu, Zijian Chen,

Organizations: Fudan University · Shanghai Innovation Institute

Causal inference in social science relies on end-to-end, intervention-centered research-design reasoning grounded in real-world policy interventions, but current benchmarks fail to evaluate this capability of large language models (LLMs). We present InterveneBench, a benchmark designed to assess such reasoning in realistic social settings. Each instance in InterveneBench is derived from an empirical social science study and requires models to reason about policy interventions and identification assumptions without access to predefined causal graphs or structural equations. InterveneBench comprises 744 peer-reviewed studies across diverse policy domains. Experimental results show that state-of-the-art LLMs struggle under this setting. To address this limitation, we further propose a multi-agent framework, STRIDES. It achieves significant performance improvements over state-of-the-art reasoning models. Our code and data are available at https://github.com/Sii-yuning/STRIDES.

Figure 1: Comparison between closed-form mathematical reasoning (Panel A) and open-ended social-science causal inference (Panel B).

Benchmark	End-to-End Causal Study Design	Real-World Reasoning	Interventional Reasoning	Structure- Agnostic Reasoning	Source	Size
GSM8K ( Cobbe et al., 2021 )	✓	✗	✗	✗	Human-written	1K
HumanEval ( Chen et al., 2021 )	✓	✗	✗	✗	Human-written	164
MMLU ( Hendrycks et al., 2021 )	✓	$\sim$	✗	✗	Professional exams	$\sim$ 14K
DiscoveryBench ( Majumder et al., 2025 )	✗	$\sim$	✗	✗	Papers + Synthetic scenarios	$\sim$ 420
QRData ( Liu et al., 2024 )	✗	✓	$\sim$	✗	Textbooks + Web + Papers	$\sim$ 701
CLADDER ( Jin et al., 2023 )	✓	✓	$\sim$	✗	Books + Papers	10K

Table 1: Comparison of benchmarks. End-to-End Causal Study Design evaluates causal reasoning as a complete social science inquiry, from problem formulation and intervention specification to causal identification and outcome interpretation. Real-World Reasoning reflects grounding in real policy interventions. Interventional Reasoning presents reasoning about interventions. Structure-Agnostic Reasoning denotes that no explicit causal graphs or structural equations are provided, requiring reasoning under open-ended institutional contexts.

Figure 2: Overview of our proposed system. The system has three stages. (1) Benchmark construction uses a Human-in-the-Loop MAS: a coordinator schedules a Paper Interpreter and Causal Designer to produce a draft causal design, a Verifier checks and routes low-quality designs for human review, and a Formatter converts approved designs into a standardized JSON schema as ground-truth designs. (2) STRIDES mirrors expert research workflows through three sequential modules. (i) The Strategic Research Design module maps unstructured metadata to statistical models using two specialized agents. (ii) The Data Environment Instantiation module employs a data retrieval agent to derive measurable indicators from model specifications and a simulation agent to generate mock data. (iii) The Code-Based Analysis and Verification module uses a code agent to produce executable statistical code and a critic agent to provide iterative feedback for design refinement, followed by a summary agent that standardizes qualified predicted designs. (3) Evaluation is performed by a separate grader module (not part of the generation pipeline), which compares model predictions against expert-verified ground truth and outputs evaluation scores.

(A) causal Inference Model Distribution
Model type	Legacy Set	Test Set
Difference-in-Differences (DiD)	241	42
Instrumental Variables (IV)	203	15
Regression Discontinuity (RD)	113	7
Synthetic Control Method (SCM)	93	7
Propensity Score Matching (PSM)	20	3

Table 2: InterveneBench statistics by causal inference model type and policy domain. We construct a time-restricted test split using 2025 papers to mitigate knowledge-cutoff contamination risks.

Model	Final Score	Model Type	Core IV	Group Def	Controls	Dep Var	Reasoning	Explanation	Improve
STRIDES(GPT-5.1)	0.665	0.515	0.691	0.544	0.842	0.974	0.695	0.650	+15.1%
w/o MAS	0.578	0.493	0.656	0.535	0.652	0.658	0.530	0.517	+15.1%
STRIDES(Claude-3.7-Sonnet)	0.653	0.650	0.728	0.601	0.570	0.892	0.620	0.347	+20.0%
w/o MAS	0.544	0.519	0.578	0.535	0.570	0.656	0.510	0.340	+20.0%
STRIDES(Claude-Sonnet-4)	0.652	0.634	0.734	0.652	0.550	0.816	0.615	0.367	+21.0%
w/o MAS	0.539	0.541	0.573	0.541	0.488	0.656	0.530	0.307	+21.0%

Table 3: Overall performance on InterveneBench. Results are reported for each vanilla LLM and its STRIDES-enhanced counterparts. The best overall result is shown in red, the best standalone LLM result is underlined . For each model pair, bold indicates the better result between the vanilla LLM and its STRIDES-enhanced counterpart. “Improve” denotes the relative final score gain of STRIDES over the corresponding vanilla LLM.

Field	Agreement	Cohen’s $\kappa$
Reasoning	85.50%	0.790
Core IV	91.60%	0.831
Group Def	89.20%	0.783
Controls	94.00%	0.880
Dep Var	83.10%	0.663
Explanation	84.30%	0.687

Table 4: Cross-model extraction consistency between GPT-5.1 and Gemini-3-Pro.

Figure 3: Case Study of Generating InterveneBench. It presents a comparison between the raw output derived from the paper ( Hammarlund and Nordin, 2025 ) generated by our original LLM-based multi-agent system and the final output refined through human expert validation.

Config	GPT-5.1	Claude-Sonnet-4	Gemini-2.5-Pro
w/o MAS	0.578	0.539	0.497
Decomposition	0.617	0.544	0.517
Decomposition	( +0.039 )	( +0.005 )	( +0.020 )
w/o Critic	0.623	0.550	0.590
w/o Critic	( +0.006 )	( +0.006 )	( +0.073 )
STRIDES	0.665	0.652	0.621

Table 5: Comparative Ablation Study. We report the overall score and the marginal gain (in parentheses).

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Model	Final Score	Model Type	Core IV	Group Def	Controls	Dep Var	Reasoning	Explanation	Improve
STRIDES(gpt-5.1)	0.665	0.515	0.691	0.544	0.842	0.974	0.695	0.650	+15.1%
Decomposition	0.617	0.540	0.631	0.619	0.638	0.820	0.540	0.497	+6.8%
w/o critic	0.623	0.527	0.654	0.594	0.634	0.870	0.525	0.570	+7.8%
w/o MAS	0.578	0.493	0.656	0.535	0.652	0.658	0.530	0.517	-
STRIDES(gpt-4.1)	0.570	0.541	0.646	0.580	0.430	0.756	0.540	0.320	+25.0%
Decomposition	0.501	0.455	0.568	0.518	0.346	0.794	0.455	0.183	+10.0%

Appendix

Table 6: Ablation results on InterveneBench. “Improve” denotes the relative final score gain of variants of STRIDES over the corresponding vanilla LLM (w/o MAS).

	Original Ground-Truth	Rewritten Ground-Truth	Output
Model Type	Regression Discontinuity (RD)	Regression Discontinuity (RD)	Sharp RD on clinical BP thresholds in panel setting
Core Independent Variable	First-ever hypertension diagnosis at wave t triggered by measured SBP $\geq$ 140 mmHg or DBP $\geq$ 90 mmHg at t-1.	First-ever hypertension diagnosis at wave t triggered by measured SBP $\geq$ 140 mmHg or DBP $\geq$ 90 mmHg at previous wave (t-1).	First-ever hypertension diagnosis at wave t based on: (i) measured SBP $\geq$ 140 mmHg or DBP $\geq$ 90 mmHg; (ii) clinician diagnosis by wave t; (iii) no prior diagnosis. RD uses distance from thresholds as running variable.
Group	Definition: First-ever diagnosis at t triggered by BP at t-1 crossing thresholds. Key RD sets (SBP-based): - Treatment: DBP $<$ 90, SBP $\geq$ 140 (local), newly diagnosed. - Control: DBP $<$ 90, SBP $<$ 140 (local), undiagnosed. - Both: No prior diagnosis; outcomes at t ( $\sim$ 3 years later). DBP-based RD: - Treatment: SBP $<$ 140, DBP $\geq$ 90, newly diagnosed. - Control: SBP $<$ 140, DBP $<$ 90, undiagnosed. Distance-to-boundary RD: - Treatment: Just on diagnosed side of (SBP, DBP) boundary. - Control: Just on non-diagnosed side.	Definition: Initial diagnosis at t triggered by BP at t-1 crossing thresholds. Primary RD (SBP-based): - Treatment: DBP $<$ 90, SBP $\geq$ 140 (local), newly diagnosed. - Control: DBP $<$ 90, SBP $<$ 140 (local), undiagnosed. - Restrictions: No prior diagnosis; outcomes at t ( $\sim$ 3 years later). DBP-based RD: - Treatment: SBP $<$ 140, DBP $\geq$ 90, newly diagnosed. - Control: SBP $<$ 140, DBP $<$ 90, undiagnosed. Distance-to-boundary RD: - Treatment: Just on diagnosed side of boundary. - Control: Just on non-diagnosed side.	Treatment: CHNS adults (age $\geq$ 18) with first-time measured SBP $\geq$ 140 or DBP $\geq$ 90 at exam wave, receiving first-ever diagnosis at that ’teachable moment’. RD sample restricted to bandwidth around cutoff(s). Control: Comparable adults with BP just below both thresholds (e.g., SBP in [140 $-$ h, 140) and DBP in [90 $-$ h, 90)), never previously diagnosed. They serve as local counterfactual for diet evolution absent informational shock.

Appendix

Table 7: Comparison among original benchmark, rewritten benchmark and vanilla LLM’s output

Metric	w/o MAS	STRIDES
Final Score	0.7456	0.7891
Model Type	0.5290	0.5600
Core IV	0.8280	0.8770
Group Def	0.7260	0.7680
Controls	0.9420	0.9960
Dep Var	0.9240	0.9780

Appendix

Table 8: GPT-5.1 results on the Legacy Set.

Model	Final Score	Model Type	Core IV	Group Def	Controls	Dep Var	Reasoning	Explanation
STRIDES (base_model)	0.5493	0.5010	0.5750	0.5260	0.4900	0.7380	0.4900	0.5267
w/o MAS (base_model)	0.4653	0.4240	0.4880	0.4450	0.4160	0.6240	0.4150	0.4467
STRIDES (CauSciBench)	0.5496	0.5029	0.5753	0.5260	0.4941	0.7334	0.4947	0.5214
w/o MAS (CauSciBench)	0.4660	0.4294	0.4947	0.4505	0.4222	0.6303	0.4197	0.3766
STRIDES (InterveneBench)	0.5548	0.5040	0.5771	0.5283	0.4953	0.7424	0.5208	0.5479
w/o MAS (InterveneBench)	0.5007	0.4820	0.5720	0.4580	0.4380	0.6260	0.4800	0.3730

Appendix

Table 9: Comparison of SFT with different training sources (Qwen3-32B). The dominant gap remains between STRIDES and w/o MAS , not between SFT sources.

$k$	Avg. Final Score	$\Delta$ vs. $k{-}1$
1	0.4150	—
2	0.4249	+2.38%
3	0.4397	+3.49%
4	0.4419	+0.50%
5	0.4419	0%
6	0.4463	+1.00%

Appendix

Table 10: Best-of-

k

test-time scaling with Kimi-K2.

Model	Rank $\Delta$	Final Score
STRIDES (GPT-5.1)	–	0.828
STRIDES (Claude-3.7-Sonnet)	–	0.639
STRIDES (GLM-4.6)	$\uparrow$ 1	0.619
STRIDES (Claude-Sonnet-4)	$\downarrow$ 1	0.611
STRIDES (Gemini-3-Flash)	$\uparrow$ 1	0.583
STRIDES (Qwen3-235B-A22B)	$\uparrow$ 3	0.560

Appendix

Table 11: Re-weighted scoring analysis.

Run	Final Score
Run 1	0.6650
Run 2	0.6580
Run 3	0.6720
Run 4	0.6510
Run 5	0.6690
Mean $\pm$ Std	0.6630 $\pm$ 0.0126

Appendix

Table 12: Run-to-run stability of GPT-5.1 + STRIDES.

Setting	Avg. Tokens	Avg. $\Delta$ Final
Vanilla	$\sim$ 3,360	–
Few-shot (3-shot)	$\sim$ 4,800	+3.3%
CoT + Self-Refine	$\sim$ 8,100	+6.1%
Best-of-6	$\sim$ 20,160	+7.5%
STRIDES	$\sim$ 19,000	+20.0%

Appendix

Table 14: Cost–performance comparison (GPT-5.1). STRIDES achieves the highest gain at a comparable token budget to Best-of-6.

Setting	Avg. total tokens per example
w/o MAS	$\sim$ 3,360
Decomposition	$\sim$ 4,000
w/o Critic	$\sim$ 10,830
STRIDES	$\sim$ 19,000

Appendix

Table 15: Token consumption per example across ablation settings.

Appendix

Table 16: Confusion matrix for Model Type predictions (STRIDES + GPT-5.1, test set,

n

=74).

Method	GPT-5.1		Claude-Sonnet-4		Gemini-2.5-pro
Method	Van.	STR.	Van.	STR.	Van.	STR.
DiD ( $n$ =42)	57.1	71.4	59.5	69.0	52.4	66.7
IV ( $n$ =15)	40.0	66.7	46.7	60.0	33.3	60.0
RD ( $n$ =7)	57.1	71.4	42.9	71.4	42.9	71.4
SCM ( $n$ =7)	42.9	71.4	28.6	57.1	42.9	57.1
PSM ( $n$ =3)	33.3	66.7	33.3	66.7	33.3	66.7

Appendix

Table 17: Model Type accuracy (%) by ground-truth causal method.

Domain	$n$	Van.	STR.	$\Delta$
Macroeconomic	17	0.541	0.622	+15.0%
Science & Technology	14	0.588	0.692	+17.7%
Social Security	13	0.592	0.701	+18.4%
Labor & Employment	10	0.563	0.672	+19.4%
Public Health	7	0.614	0.728	+18.6%
Housing & Urban	5	0.570	0.658	+15.4%

Appendix

Table 18: Per-domain Final Score for GPT-5.1.

Causal reasoning in language models studies how models identify cause-and-effect structure, predict interventions, and answer counterfactual questions.

See all 102 papers

Why LLMs Fail at Causal Discovery and How Interventional Agents Escape
May 26, 2026Amartya Roy, Sonali ParbhooCausal Discovery Causal Reasoning in Language Models
SIRE, IIT Delhi and Robert Bosch GmbH, India · Imperial College London
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
Feb 24, 2026Ayush Sawarni, Jiyuan Tan, Vasilis SyrgkanisCausal Effect Estimation Benchmark Design
Stanford University
CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Jul 9, 2026Andrej Leban, Theo Saarinen, Yuekai SunStructural Causal Models AI Agent Benchmarks
Department of Statistics, University of Michigan, Ann Arbor, MI, United States

May 26, 2026J/K move · Enter open · S save

Amartya Roy, Sonali Parbhoo

SIRE, IIT Delhi and Robert Bosch GmbH, India · Imperial College London

Causal Discovery Causal Reasoning in Language Models Causal Identifiability Causal Interventions in Language Models

Open paper AskPDF arXiv

Causal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it reliably remains an open question. Recent benchmarks show that even fine-tuned models plateau on simple causal graphs and degrade as complexity grows, but why they fail has not been established. We prove the failure is fundamental: supervised fine-tuning, direct preference optimization, and in-context learning all produce predictors that cannot distinguish between causal graphs generating similar observational data, and any attempt to do so requires the model's internal representations to grow unboundedly, violating the very conditions under which these methods work. We formalize this as a kernel obstruction theorem, establishing that the limitation is intrinsic to the learning paradigm, not any particular model or dataset. We propose Agentic Causal Bayesian Optimization (A-CBO), wherein a frozen language model serves as an interventional oracle answering targeted queries about intervention effects, while an external Bayesian loop concentrates beliefs over candidate graphs in logarithmically many rounds. Because the decision operates outside the space where the obstruction applies, A-CBO provably converges while the underlying model remains unchanged. On Corr2Cause, A-CBO matches fine-tuned baselines without any training. On Extended Corr2Cause, a new benchmark scaling to 24 variables with 18K test samples, A-CBO significantly outperforms both fine-tuning and preference optimization, with the advantage growing

True $\backslash$ Pred