Active Causal Discovery Benchmark: Evaluating LLM Agents Under Budgeted Interventions

Sagar Deb; Devam Shah; Ashwanth Krishnan

cs.LGSep 16, 2026

Active Causal Discovery Benchmark: Evaluating LLM Agents Under Budgeted Interventions

Authors: Sagar Deb, Devam Shah, Ashwanth Krishnan

Organizations: QpiAI

Abstract

We introduce the Active Causal Discovery Benchmark (ACDB), an SCM-grounded environment for evaluating whether LLM agents recover causal graph structure from observations and budget-constrained hard interventions. ACDB pairs a linear-Gaussian world generator with a fixed observe-intervene-submit API and a three-layer scoring contract that separates skeleton recovery, DAG recovery, and intervention efficiency. On the current six-level ladder, PC with a greedy active orientation heuristic is the strongest non-oracle method (directed F1 42.7%, SHD 4.79), ahead of Claude Sonnet 4.6 raw active (31.7%, 7.25) and GPT-5.4 raw active (22.9%, 9.27). The most informative diagnostic is the precision-recall decomposition: PC under-commits with high precision, LLMs over-commit with lower precision, and statistical-tool access often increases abstention rather than useful intervention. A structure-blind random DAG baseline reaches 23.6% directed F1 on this dense v0 ladder; a density probe lowers this floor to 16.9%, motivating the v1 calibration pass. The current results should therefore be read as a benchmark audit and calibration report, not as evidence that current LLMs solve active causal discovery.

Figures & tables

Figure 1: Observation identifies a CPDAG, not necessarily the full DAG. In this example the collider into

X_{4}

is compelled from observational structure, while the two incident edges at

X_{1}

remain undirected until an intervention on

X_{1}

resolves them.

Figure 2: ACDB instance generation. Each accepted instance is produced by sampling a graph, converting it to its CPDAG, rejecting invalid or near-unfaithful cases, parameterizing a linear–Gaussian SCM, computing the optimal intervention set, anonymizing labels, and sampling data. Rejection gates loop back to graph sampling.

Figure 3: ACDB fixes the episode protocol and separates adjacency, orientation, and budget-use scores.

Level	Role	$d$	$k$	$n_{\mathrm{obs}}$	$n_{\mathrm{int}}$	$\sigma^{2}$	slack
0	tutorial	4	5	50	25	0.5	2
1	standard	5	6	25	15	1.0	1
2	statistical	5	6	15	10	1.5	1
3	structural	7	9	25	15	1.0	1
4	pressure	5	6	25	15	1.0	0
5	hard	7	9	15	10	1.5	0

Table 1: ACDB ladder. Each level fixes

d

(variables),

k

(true edge count),

n_{\mathrm{obs}}

n_{\mathrm{int}}

(sample counts),

\sigma^{2}

(noise variance), and slack (interventions above the precomputed optimal). The intervention budget is

B=|\mathcal{I}^{\star}|+\texttt{slack}

and is therefore instance-specific.

Figure 4: Density leakage in the structure-blind random baseline. Points are per-level directed F1; tick labels show graph density

\rho

for the v0 ladder and density probe. Lowering density reduces the overall random floor from

23.6\%

16.9\%

Figure 5: Headline DAG-recovery performance on the v0 active ladder. PC+greedy is the strongest non-oracle method, Sonnet raw is the strongest LLM policy, and several LLM settings sit close to the structure-blind random floor.

Observational panel (no interventions)
Method	Model	Skel F1	Dir P	Dir R	Dir F1	SHD	$\eta$
PC	–	62.1	82.6	10.0	13.6	6.25	–
LLM (raw)	GPT-5.4	65.7	74.1	19.2	22.6	9.40	–
LLM (stats)	GPT-5.4	35.9	68.4	9.2	13.3	6.75	–
LLM (raw)	Sonnet 4.6	57.7	38.9	34.5	36.1	7.46	–
LLM (stats)	Sonnet 4.6	60.7	76.9	12.5	15.0	9.92	–

Table 2: Headline results as weighted averages across ladder levels L0–L5. Skel F1 is against the true CPDAG; Dir P, Dir R, Dir F1 are against the true DAG; SHD is edge edit distance (reversal=1);

\eta

is intervention efficiency against the precomputed optimal set. All values in %; SHD is a mean edge count. Oracle submits the true DAG; PC + greedy is the strongest non-oracle active method. Directed precision and recall are reported separately because LLM and PC errors lie on different sides of the P/R trade-off (§ 5 ).

Observational panel
Method	Model	L0	L1	L2	L3	L4	L5	Overall
PC	–	13.9	23.8	3.1	17.2	20.0	3.8	13.6
LLM (raw)	GPT-5.4	39.8	27.7	7.5	18.8	24.4	17.7	22.6
LLM (stats)	GPT-5.4	12.3	13.4	9.4	11.4	21.9	11.8	13.3
LLM (raw)	Sonnet 4.6	26.9	34.9	39.3	33.8	41.7	40.1	36.1
LLM (stats)	Sonnet 4.6	24.7	14.0	6.2	10.9	22.4	12.0	15.0

Table 3: Directed F1 per ladder level, by panel. Weighted averages in the rightmost column match Table 2 . Values are means over

8

preflighted seeds. The random row is structure-blind and uses only

d

. All values in %.

Causal Discovery

Causal discovery infers cause-and-effect structures from observational and interventional data, distinguishing causal relationships from correlation.

See all 221 papers

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
May 25, 2026Junlin Yang, Dylan Zhang, Xiangchen Song +7Structural Causal Models LLM Agent Evaluation
Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University +2
Why LLMs Fail at Causal Discovery and How Interventional Agents Escape
May 26, 2026Amartya Roy, Sonali ParbhooCausal Discovery Causal Reasoning in Language Models
SIRE, IIT Delhi and Robert Bosch GmbH, India · Imperial College London
Benchmarking and Enhancing Agentic Causal Discovery
Oct 16, 2026Pingchuan Ma, Rui Ding, Shuai WangCausal Discovery AI Agent Benchmarks
Zhejiang University of Technology · Microsoft · The Hong Kong University of Science and Technology

May 25, 2026J/K move · Enter open · S save

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

Junlin Yang, Dylan Zhang, Xiangchen Song, Qirun Dai+6

Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University+2

Structural Causal Models LLM Agent Evaluation Causal Discovery AI Agent Benchmarks Causal Reasoning in Language Models

Open paper AskPDF arXiv

We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge

F_1

. Mixed observation-intervention strategies improve structural fidelity, while pure intervention remains difficult even for strong agents. We identify premature stopping as a major weakness and show that consistency verification mitigates it. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents' limits as experimental causal reasoners.

Abstract

Causal Discovery

Explore similar work

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

Why LLMs Fail at Causal Discovery and How Interventional Agents Escape

Benchmarking and Enhancing Agentic Causal Discovery