Benchmarking and Enhancing Agentic Causal Discovery

Pingchuan Ma; Rui Ding; Shuai Wang

cs.LGOct 16, 2026

Benchmarking and Enhancing Agentic Causal Discovery

Authors: Pingchuan Ma, Rui Ding, Shuai Wang

Organizations: Zhejiang University of Technology · Microsoft · The Hong Kong University of Science and Technology

Abstract

LLM agents can now run causal discovery algorithms on a dataset and report what comes out. Whether they do more than that, such as noticing where an algorithm went wrong and fixing it with further tests, is hard to tell from current benchmarks. Most of them give the variables meaningful names, and an LLM shown only the names of the Tübingen cause-effect pairs already gets 99% of the directions right. With the names removed, the data are usually clean enough that picking a better algorithm than PC closes most of the gap. We introduce Heuresis, a benchmark of anonymous simulated datasets whose mechanisms range from linear Gaussian to heavy-tailed. For each family of datasets it separates what a better existing algorithm can gain from what only a closer look at the data can gain, and it traces every edge an agent changes. Every family leaves a gap that none of the algorithms, the router and the two causal-discovery foundation models we ran can close, so a score above the best chosen algorithm has to come from using the data. We instantiate the benchmark with an agent harness and three LLMs, GPT-6.1 Sol, Claude Opus 5.5 and Qwen 3.8 Flash. GPT-6.1 Sol surpasses the best chosen algorithm on all six families, by 14% adjacency F1 on the most difficult one, and Claude Opus 5.5 does so by 7.4% there. A third of the true edges the two agents restore on the two hardest families appear in no algorithm's output. The code and data will be publicly released to facilitate future work.

Figures & tables

Figure 1: Overview of Heuresis .

Figure 2: Running example of PC.

Figure 3: A Heuresis episode and its evaluation.

Benchmark	D1 Anonymous	D2 Headroom	D3 Gains	D4 Decomposed
Corr2Cause	✓	$\times$	$\times$	$\times$
CausalBN-Bench	$\times$	$\times$	$\times$	$\times$
chatPC	$\times$	$\times$	$\times$	$\sim$
CausalArena	–	$\times$	$\times$	$\times$
ACDB	✓	$\times$	$\times$	$\times$
Heuresis (ours)	✓	✓	✓	✓

Table 1: Existing benchmarks against the design requirements D1–D4, as described in their papers. ✓ yes,

\sim

partly,

\times

no, – not reported.

nonlinear

input-dep. noise

non-monotone

t_{3}

noise

difficulty

Family

linear

MLP

piecewise

sigmoid

MLP

sigmoid

quadratic

|\cdot|

Fourier

linear

Wrong

Zero

Headroom

⚫

2.4

23%

.121

anm

⚫

1.4

26%

.078

hetero

⚫

1.0

39%

.049

wild

⚫

3.0

.169

mixed

⚫

1.2

36%

.059

Table 2: Mechanism families of Heuresis

Figure 4: Running example of Heuresis instance.

Figure 5: The working graph over an episode.

	lg	anm	hetero	wild	mixed	gauntlet
BCA ϕ	.921	.906	.934	.814	.920	.849
Oracle	.973	.984	.982	.977	.986	.983
$H^{\mathrm{s}}_{\phi}$	.069	.000	.000	.006	$-$ .007	.000
$H^{\mathrm{r}}_{\phi}$	.052	.078	.049	.163	.066	.134
$\Delta^{\mathrm{r}}_{\phi}(\texttt{router})$	$-$ .029	.000	.000	$-$ .006	.007	.000
$\Delta^{\mathrm{r}}_{\phi}(\texttt{CDFM})$	$-$ .186	$-$ .190	$-$ .177	$-$ .104	$-$ .191	$-$ .124

Table 3: F1 of the best chosen algorithm (BCA ϕ ) and the oracle, the two headrooms, and the residual gain

\Delta^{\mathrm{r}}_{\phi}

of three baselines.

Figure 6: Residual gain on Heuresis of LLM agents.

	GPT-6.1 Sol				Claude Opus 5.5
	PC-KCI	merge ∗	agent	unsupp.	PC-KCI	merge ∗	agent	unsupp.
lg	.854	.892	.978	0/72	.838	.884	.972	8/61
anm	.887	.887	.958	4/39	.891	.891	.910	2/29
hetero	.914	.920	.967	5/29	.913	.922	.940	3/24
wild	.813	.813	.935	20/57	.817	.817	.885	13/40
mixed	.919	.933	.972	4/34	.915	.928	.951	0/28

Table 4: Trace attribution. merge ∗ is the best F1 reached by adding to the PC-KCI output every edge that at least

k

of the other algorithms the agent called propose, with

k

chosen per family in hindsight. Unsupp. : true edges missing from the PC-KCI output that the agent restores although no algorithm it called proposes them, out of all such edges it restores.

Figure 7: (a) Recall of the wrongly deleted edges. (b) Gain in F1 of the guided episode over the unguided episode on the same instance.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Reference	Scale	Result
Graph primitives
d-separation	two independent implementations (moralized ancestral graph; path blocking)	$2.16\times 10^{6}$ random DAGs, $4.3\times 10^{8}$ queries	0 disagreements
DAG to CPDAG	brute-force enumeration of the Markov equivalence class; Chickering’s edge labeling; causal-learn	$3.4\times 10^{6}$ DAGs (enumeration), $5.8\times 10^{6}$ DAGs (labeling, causal-learn)	0 disagreements
Exhaustive $d\leq 6$	causal-learn: d-separation, CPDAG and oracle PC	all $3{,}811{,}355$ labeled DAGs (equals OEIS A003024)	0 disagreements
PC
Oracle PC	true CPDAG returned, first-set and union rules	$6\times 10^{5}$ random DAGs plus the exhaustive set	0 failures

Appendix

Table 5: Instrument verification checks and results, grouped by the component under test.

Figure 8: Scripted use of the deletion log of PC-KCI. (a) AUC of each signal for ranking the wrongly deleted edges among all deletions; the first two signals need no new tests. (b) Change in adjacency F1 from restoring edges by each signal, compared with the ideal oracle that restores every wrongly deleted edge. 32 test instances per family. Error bars represent bootstrapped 95% confidence intervals.

	GPT-6.1 Sol		Claude Opus 5.5		Qwen 3.8 Flash
	F1	$\Delta^{\mathrm{r}}_{\phi}$	F1	$\Delta^{\mathrm{r}}_{\phi}$	F1	$\Delta^{\mathrm{r}}_{\phi}$
lg	.978	.043 [.019, .070]	.972	.037 [.013, .066]	.945	.009 [ $-$ .018, .039]
anm	.958	.052 [.032, .074]	.906	$-$ .000 [ $-$ .027, .027]	.899	$-$ .008 [ $-$ .042, .027]
hetero	.967	.042 [.027, .058]	.934	.010 [ $-$ .012, .031]	.903	$-$ .022 [ $-$ .048, .001]
wild	.935	.115 [.083, .146]	.880	.060 [.030, .090]	.843	.023 [ $-$ .013, .059]
mixed	.972	.045 [.028, .065]	.952	.025 [ $-$ .000, .051]	.924	$-$ .003 [ $-$ .033, .028]

Appendix

Table 6: Agent performance (every episode submitted a graph) on 32 test instances per family: adjacency F1 and the paired difference from the family’s best chosen algorithm. Rows marked (all) are over all families. GPT-6.1 Sol and Claude Opus 5.5 are shown in repeat 1; their repeats 2 and 3, and the finished-only and all-episode views of the runs without a retry in the last turn, are in Table 13 .

Family	$n$	adjacency F1	nSHD	$n$	adjacency F1	nSHD
	GPT-6.1 Sol			Claude Opus 5.5
lg	32	.125 [.086, .166]	.068 [.047, .091]	23	.134 [.080, .189]	.079 [.050, .108]
anm	32	.071 [.047, .097]	.110 [.091, .128]	30	.018 [ $-$ .014, .048]	.088 [.065, .111]
hetero	32	.053 [.035, .072]	.086 [.069, .103]	26	.027 [.002, .053]	.039 [.014, .062]
wild	32	.122 [.091, .152]	.135 [.115, .156]	30	.068 [.039, .097]	.111 [.090, .131]
mixed	32	.053 [.035, .071]	.092 [.069, .114]	30	.035 [.007, .064]	.077 [.053, .101]

Appendix

Table 7: Agent gain over the in-episode output of PC-KCI (GES on the last row of each model), scored by the same scorer on the same instance, repeat 1, mean paired gain with 95% bootstrap intervals over instances; positive is better for the agent, and nSHD is reported as its reduction.

n

is the number of episodes in which the agent called the algorithm (all 32 for GPT-6.1 Sol). Bold: the interval excludes zero.

Toolbox	Family	Precision	Recall	vs PC-KCI	Guided $-$ unguided
GPT-6.1 Sol
full	lg	.88 [.77, .96]	.94 [.89, .99]	.112 [.074, .149]	$-$ .006 [ $-$ .019, .004]
	anm	.87 [.78, .95]	.76 [.63, .88]	.048 [.028, .072]	$-$ .003 [ $-$ .016, .008]
	hetero	.89 [.79, .98]	.80 [.68, .91]	.039 [.023, .055]	$-$ .004 [ $-$ .013, .006]
	wild	.98 [.94, 1.00]	.70 [.59, .80]	.125 [.093, .156]	.009 [ $-$ .007, .028]
minimal	lg	.83 [.73, .93]	.90 [.84, .96]	.091 [.057, .125]	$-$ .027 [ $-$ .051, $-$ .006]

Appendix

Table 8: Guided condition (32 test instances per family, all episodes): precision and recall of restoring the wrongly deleted edges of PC-KCI, the paired gain over PC-KCI, and the paired gain over the agent’s own unguided run with the full toolbox on the same instances (repeat 1 for GPT-6.1 Sol and Claude Opus 5.5), with 95% bootstrap intervals over episodes.

GPT-6.1 Sol
		lg		wild
Factor	Level	$\Delta^{\mathrm{r}}_{\phi}$	shortcut	$\Delta^{\mathrm{r}}_{\phi}$	shortcut	wall
prompt	neutral	.049 [.028, .075]	.21	.101 [.078, .125]	.00	.00
	scoring	.050 [.029, .076]	.23	.104 [.081, .128]	.02	.00
	warning	.053 [.030, .079]	.19	.095 [.076, .115]	.00	.00
	rule	.044 [.021, .070]	.12	.104 [.083, .125]	.02	.00

Appendix

Table 9: Prompt level (each level adds to the previous one: neutral task statement; budgets and the scoring statement; a warning that standard algorithms can fail under nonlinear or heteroscedastic mechanisms; PC’s deletion rule) and stated target (first 16 of the 32 test instances per family, each row averaged over the other factor; GPT-6.1 Sol and Claude Opus 5.5 in repeat 1): paired gain in adjacency F1 over the best chosen algorithm, share of episodes whose final graph is PC’s output, and share of episodes that reached the wall.

GPT-6.1 Sol
Factor	Level	$\Delta^{\mathrm{r}}_{\phi}$ , lg	$\Delta^{\mathrm{r}}_{\phi}$ , wild	wall	unsubmitted	tokens
tokens per turn	4 096	.044 [.019, .074]	.112 [.084, .141]	.00	.00	7 k
	8 000	.043 [.017, .072]	.094 [.070, .121]	.00	.00	7 k
	16 000	.051 [.026, .080]	.099 [.075, .123]	.00	.00	6 k
	32 000	.045 [.016, .076]	.109 [.080, .142]	.00	.00	6 k
wall	1 200 s	.049 [.030, .071]	.107 [.088, .128]	.00	.00	7 k

Appendix

Table 10: Budget sweep (first 16 of the 32 test instances per family, each row averaged over the other factor; GPT-6.1 Sol and Claude Opus 5.5 in repeat 1): paired gain in adjacency F1 over the best chosen algorithm, share of episodes that reached the wall and submitted in the retry, share left unsubmitted, and completion tokens per episode.

Family	empty	complete	PC-KCI	random, oracle’s count	oracle
lg	.000	.361	.861	.811 [.773, .848]	.970
anm	.000	.363	.906	.857 [.830, .884]	.984
hetero	.000	.362	.928	.892 [.871, .913]	.981
wild	.000	.362	.818	.742 [.710, .774]	.973
mixed	.000	.378	.934	.899 [.867, .928]	.987
gauntlet	.000	.340	.849	.768 [.738, .799]	.990

Appendix

Table 11: Trivial floors (adjacency F1, 32 test instances per family): the empty graph, the complete graph (every deletion of PC-KCI restored), PC-KCI itself (nothing restored), a random restoration of as many deleted pairs as the oracle restores (20 draws per instance), and the oracle.

Figure 9: Qwen 3.8 Flash on wild under the three last-turn settings of Table 12 (no last turn, one last turn in which it can only submit, the same turn with a retry), scored over all episodes and over finished episodes only. Intervals as in Figure 6 .

Item	Value
Models	GPT-6.1 Sol (OpenAI), reasoning effort “medium”; Claude Opus 5.5 (Anthropic), adaptive thinking at effort “high”; Qwen 3.8 Flash (Alibaba), thinking mode on. Main runs of 2026-10-09; guided, prompt and budget runs of GPT-6.1 Sol and Claude Opus 5.5 of 2026-10-10
Prompt	neutral level, no stated target; no algorithm is named below the rule level (Table 9 )
Toolbox	the full toolbox: algorithm menu, conditional-independence tests (single, batched and subsampling stability), code analysis, local score, orientation probes, data summary, graph editor, and submission of the final edge list
Budgets	1000 CI tests; 60 turns; 16000 output tokens per turn; 2400 s wall; 12 menu calls; all stated to the model; the tokens per turn and the wall are swept in Table 10
Last-turn setting	no last turn (repeat 1); one last turn in which the agent can only submit (repeat 2); the same turn with tolerant reading of the edge list and one retry (later runs)
Runs per cell	32 test instances per family. GPT-6.1 Sol and Claude Opus 5.5: repeats 1 to 3 on lg , anm , hetero and wild , repeat 1 on mixed and gauntlet ; Qwen 3.8 Flash: the same cells as the other two models in Table 13 . Guided, prompt and budget runs: repeat 1 for every model (32 instances per family guided, 16 per family in the prompt and budget sweeps)

Appendix

Table 12: Harness disclosure for the agent results.

				all episodes		finished
Family	Repeat	Last turn	Wall / rescued	F1	$\Delta^{\mathrm{r}}_{\phi}$	$n$	F1	$\Delta^{\mathrm{r}}_{\phi}$
GPT-6.1 Sol (reasoning effort “medium”)
lg	1	retry	0 / 0	.978	.043 [.019, .070]	32	.978	.043 [.019, .070]
	2	retry	0 / 0	.983	.048 [.021, .078]	32	.983	.048 [.021, .078]
	3	retry	0 / 0	.983	.048 [.021, .077]	32	.983	.048 [.021, .077]
anm	1	retry	0 / 0	.958	.052 [.032, .074]	32	.958	.052 [.032, .074]

Appendix

Table 13: Per-repeat results for the main agent condition. F1 is adjacency F1;

\Delta^{\mathrm{r}}_{\phi}

is the paired difference from the family’s best chosen algorithm. Repeats 1 to 3 on four families and repeat 1 on mixed and gauntlet , the same for the three models. Last turn: none, one turn in which the agent can only submit, or that turn with a retry.

Causal Discovery

Causal discovery infers cause-and-effect structures from observational and interventional data, distinguishing causal relationships from correlation.

See all 221 papers

Active Causal Discovery Benchmark: Evaluating LLM Agents Under Budgeted Interventions
Sep 16, 2026Sagar Deb, Devam Shah, Ashwanth KrishnanCausal Discovery LLM Agent Evaluation
QpiAI
CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
May 25, 2026Junlin Yang, Dylan Zhang, Xiangchen Song +7Structural Causal Models LLM Agent Evaluation
Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University +2
Causal Discovery in the Era of Agents
Jun 22, 2026Yujia Zheng, Vishal Verma, Mantej Gill +3AI Agents for Scientific Discovery Causal Discovery
1Carnegie Mellon University. · 2Mohamed bin Zayed University of Artificial Intelligence.

Sep 16, 2026J/K move · Enter open · S save

Active Causal Discovery Benchmark: Evaluating LLM Agents Under Budgeted Interventions

Sagar Deb, Devam Shah, Ashwanth Krishnan

QpiAI

Causal Discovery LLM Agent Evaluation Structural Causal Models AI Agent Benchmarks

Open paper AskPDF arXiv

We introduce the Active Causal Discovery Benchmark (ACDB), an SCM-grounded environment for evaluating whether LLM agents recover causal graph structure from observations and budget-constrained hard interventions. ACDB pairs a linear-Gaussian world generator with a fixed observe-intervene-submit API and a three-layer scoring contract that separates skeleton recovery, DAG recovery, and intervention efficiency. On the current six-level ladder, PC with a greedy active orientation heuristic is the strongest non-oracle method (directed F1 42.7%, SHD 4.79), ahead of Claude Sonnet 4.6 raw active (31.7%, 7.25) and GPT-5.4 raw active (22.9%, 9.27). The most informative diagnostic is the precision-recall decomposition: PC under-commits with high precision, LLMs over-commit with lower precision, and statistical-tool access often increases abstention rather than useful intervention. A structure-blind random DAG baseline reaches 23.6% directed F1 on this dense v0 ladder; a density probe lowers this floor to 16.9%, motivating the v1 calibration pass. The current results should therefore be read as a benchmark audit and calibration report, not as evidence that current LLMs solve active causal discovery.

Abstract

Causal Discovery

Explore similar work

Active Causal Discovery Benchmark: Evaluating LLM Agents Under Budgeted Interventions

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

Causal Discovery in the Era of Agents