When Should Semantic Priors Enter Partially-Identified Causal Discovery?
Authors: Muhammad Ahmed Mohsin, Saad Bin Ather, Valentin Six, Lancelot Da Costa, Tommaso Salvatori, Bernhard Scholkopf
Organizations: Stanford University · FAST-NUCES Lahore, Pakistan · University of Virginia Charlottesville, USA · VERSES Technologies Inc. · Relational Intelligence · ELLIS Institute
Observational causal discovery often leaves causal di- rections unresolved or unstable. Large language mod- els (LLMs) can provide semantic information to resolve these directions, but incorrect proposals can exclude the true causal graph and invalidate downstream inference. We study three separate decisions: where to allocate semantic queries, which proposals to admit, and which causal graphs to retain. Our framework distinguishes population non-identifiability from finite-sample insta- bility, evaluates semantic orientation against statistical and random controls, and examines calibrated admis- sion and error-tolerant DAG retention. Experiments across benchmark networks show that semantic infor- mation improves recovery over coin-flip orientations, but targeting unstable edges has no established ad- vantage over matched-random allocation. The tested admission certificate cannot support nontrivial admis- sion with the available calibration tasks. Moreover, downstream retention depends on reference adequacy: learned references contain the true DAG in only three of 80 planned tasks. These findings separate the value of semantic information from the reliability of its alloca- tion, admission, and retention, and identify calibration size and reference adequacy as key limitations.
Figures & tables
Figure 1: Semantic assistance in causal discovery. Allocation selects queries, admission checks proposals, and retention preserves plausible DAGs.
Figure 2: Semantic information, allocation, and resolution. (a) Semantic-minus-coin, coin-minus-Base, and semantic-minus-Base F1-only contrasts. (b) F1+F2 minus matched random in the four primary settings. (c) F1+F2 minus bootstrap-only F2 resolution in the four primary settings. (d) Allocation and resolution contrasts on the additional families. Bars show equal-network effects; whiskers show exploratory 95% network intervals. Primary contrasts use five networks; additional-family contrasts use three. Panels use different scales. All displayed allocation intervals include zero.
PC
BOSS
Policy
500
2000
500
2000
Base
.4011
.4962
.6256
.6500
F1-only
.6177
.6831
.7744
.8414
F1+rand
.6176
.7179
.7831
.8450
F1+F2-data
.6206
.6837
.7662
.8405
Bootstrap–LLM
.6215
.6867
.7685
.8488
Table 1: Primary graph recovery and internal baselines. Pooled Directed-F1 point estimates; usable counts are 25/20/25/25 in column order. The table provides descriptive recovery comparisons. Exploratory 95% intervals for the primary paired, equal-network contrasts appear in Figure 2 . These contrasts use different weighting from the pooled means.
Figure 3: Baseline comparisons. (a) Pooled Directed-F1 changes relative to F1+F2 for three internal alternatives across the four primary settings. (b) Native Guess2Graph Skeleton-F1 improvement over its compatible PC baseline; this is not a matched directional comparison. Propagation is a frozen-response replay with archived responses available for 260/285 selected edges. Both panels report descriptive point estimates without uncertainty intervals; their metrics and comparison populations differ.
Figure 4: Semantic content and representation controls. (a) Raw directional accuracy uses 1,280 truth-adjacent requests per primary condition. (b) UNKNOWN uses all 1,760 completed requests per condition. (c) Order contradiction uses only pairs with two parsed directions; denominators are printed at each point. (d) Additional-family raw accuracy uses 380 requests per condition and representation. Primary temperature is .8 ; additional-family temperature is 0 . Values are pooled descriptive proportions, without independent-request error bars. These panels do not share a statistical denominator.
Target risk α
.10
.20
.30
Hoeffding, zero loss
205
52
23
Exact, zero loss events
39
19
12
F1 available
10 graph tasks
F2 available
9 graph tasks
Hoeffding selector
Complete abstention
Table 2: Calibration feasibility. Independent-task requirements with six fixed thresholds and η=.1 . The exact zero-event row bounds the event of any task loss; it is a theoretical comparison, not a certified real-data result. Positive empirical loss raises the Hoeffding requirement.
Figure 5: Retention depends on reference adequacy. Under the Oracle reference, calibrated tolerance restores exact truth retention while preserving effect coverage. Under the learned X6 reference, exact truth retention remains low because the stored reference usually excludes the true DAG. The panels use different evaluation populations and are descriptive.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Network
Nodes
Edges
Recovery panel
Asia
8
8
Primary
Sachs
11
17
Primary
Child
20
25
Primary
Insurance
27
52
Primary
Alarm
37
46
Primary
Barley
48
84
Additional
Appendix
Table 3: Benchmark structures. Counts describe the benchmark DAGs.
Figure 6: Fitted-edge composition. Fraction of fitted skeleton edges that are unresolved ( F1 ), selected as bootstrap-unstable ( F2 ), or other committed edges. Values are averaged within networks and then across networks equally.
Policy
PC-500
PC-2000
BOSS-500
BOSS-2000
Base
.4011 / 23.44
.4962 / 20.75
.6256 / 15.00
.6500 / 14.88
F1-only
.6177 / 17.60
.6831 / 15.50
.7744 / 10.08
.8414 / 8.24
F1+rand
.6176 / 17.68
.7179 / 14.75
.7831 / 10.04
.8450 / 8.20
F1+F2-data
.6206 / 17.60
.6837 / 15.55
.7662 / 10.32
.8405 / 7.92
Bootstrap–LLM
.6215 / 17.88
.6867 / 15.75
.7685 / 10.76
.8488 / 8.24
Propagation
.6184 / 17.36
.7064 / 14.95
.7811 / 9.96
.8462 / 8.12
Appendix
Table 4: Primary recovery values. Each cell is Directed-F1 / SHD. Pooled usable counts are 25/20/25/25 in column order. Bootstrap–LLM and Propagation are the additional internal baselines reported in the main text.
Baseline
Initial
Intermediate
Final
Bootstrap–LLM
285 eligible
141 agree
132 admitted
Propagation
285 selected
260 responses
25 missing
Appendix
Table 5: Internal baseline diagnostics. Counts summarize the proposal or replay populations used by the two additional internal baselines. Propagation response availability is reported separately from its selected query budget.
Contrast
Setting
Pooled F1
Equal-network F1
95% interval
Pooled SHD
F2 minus random
PC-500
+.01672
+.01672
[−.02986,.06330]
−.60
PC-2000
−.01479
−.01158
[−.03972,.01656]
+.25
BOSS-500
−.00379
−.00379
[−.03174,.02416]
.00
BOSS-2000
+.01596
+.01596
[−.00526,.03718]
−.76
Semantic minus data
PC-500
+.01375
+.01375
[−.01759,.04510]
−.52
PC-2000
+.01940
+.02406
[−.03006,.07819]
−.55
Appendix
Table 6: Primary mechanism contrasts. Pooled and equal-network Directed-F1 differ when usable seed counts differ. Intervals are exploratory 95% network intervals.
Contrast
Estimate
95% interval
Semantic minus coin
+.08263
[.01917,.14609]
Coin minus Base
+.10363
[.01250,.19476]
Semantic minus Base
+.18626
[.05838,.31413]
Appendix
Table 7: F1 coin-flip control. Equal-network Directed-F1 contrasts across 95 eligible tasks and five networks.
Network
PC-500
PC-2000
BOSS-500
BOSS-2000
F1+F2 minus F1+rand
Asia
+.0214
−.0500
.0000
.0000
Sachs
.0000
−.0004
−.0396
+.0125
Child
−.0091
−.0106
+.0160
.0000
Insurance
−.0088
−.0057
+.0138
+.0304
Alarm
+.0801
+.0089
−.0092
+.0369
Appendix
Table 8: Network heterogeneity. Mean paired Directed-F1 over available seeds. The upper block tests allocation; the lower block tests semantic resolution. Entries retain the precision of the archived table.
Figure 7: Network heterogeneity of the F2 contrasts. Grouped horizontal bars show (a) F1+F2 minus matched random and (b) F1+F2 minus bootstrap-only resolution. Rows identify networks; colors identify estimator and sample-size settings. Bars show descriptive within-network Directed-F1 differences, without network-specific intervals.
Cap
PC-500
PC-2000
BOSS-500
BOSS-2000
0
.0000 / .00
.0000 / .00
.0000 / .00
.0000 / .00
10
.0000 / .00
.0000 / .00
.0000 / .00
.0000 / .00
20
−.0057 / +.04
.0000 / .00
.0000 / .00
.0000 / .00
40
−.0067 / +.08
−.0202 / +.20
−.0079 / +.12
.0000 / .00
80
+.0054 / −.08
−.0116 / +.15
−.0077 / +.12
+.0056 / −.08
Full
+.0167 / −.60
−.0148 / +.25
−.0038 / .00
+.0160 / −.76
Appendix
Table 9: Pooled allocation contrasts over response budgets. Each cell is Δ Directed-F1 / Δ SHD for F1+F2 minus F1+rand. Full is a categorical endpoint.
Figure 8: Allocation across response budgets. Pooled paired Directed-F1 differences for F1+F2 minus F1+rand. Lines connect numerical response-request caps for PC and BOSS at N=500,2000 . With R=10 , caps 10/20/40/80 permit at most 1/2/4/8 queried edges. Full allocation is an unconnected categorical endpoint. No uncertainty intervals are supplied for this sweep.
Contrast
Population
Networks
Effect
95% interval
F2 minus random
Primary
5
+.00474
[−.01527,.02475]
Additional
3
−.00104
[−.00605,.00398]
Combined sensitivity
8
+.00257
[−.00795,.01310]
Semantic minus data
Primary
5
+.01718
[−.00515,.03951]
Additional
3
+.00418
[.00243,.00593]
Combined sensitivity
8
+.01230
[−.00038,.02499]
Appendix
Table 10: Additional-family and mixed-protocol summaries. Values are equal-network Directed-F1 effects. The combined row is a heterogeneous-protocol sensitivity, not one homogeneous eight-network experiment.
Condition
Raw correct
Valid-only correct
UNKNOWN
Contradiction
Named
1143/1280
1143/1230
146/1760
32/774
No-Benchmark
1114/1280
1114/1231
132/1760
50/783
Anonymous
1052/1280
1052/1214
139/1760
79/778
Shuffled
705/1280
705/963
538/1760
95/522
Appendix
Table 11: Primary semantic-control counts. Raw accuracy and valid-only accuracy use truth-adjacent requests. UNKNOWN includes 480 nonadjacent controls among 1,760 requests. Contradiction uses only presentation pairs yielding two parsed directions. These are different populations.
Condition
Representation
Raw correct
Valid-only correct
UNKNOWN
Contradiction
Original
Canonical
305/380
305/355
25/380
20/175
Original
Permuted
316/380
316/360
20/380
24/180
Anonymous
Canonical
245/380
245/335
45/380
35/150
Anonymous
Permuted
242/380
242/325
55/380
31/150
Shuffled
Canonical
141/380
141/256
124/380
20/110
Shuffled
Permuted
124/380
124/263
117/380
21/119
Appendix
Table 12: Additional-family response counts. Each condition–representation cell contains 380 truth-adjacent requests at temperature 0 . Contradiction has its own paired denominator.
Policy
Pool
Admission
Task loss
Add-one
F1
1.000
.162
Add-one
F2
.805
.175
Agreement
F1
.944
.170
Agreement
F2
.965
.198
Appendix
Table 13: Descriptive empirical selection at α=.20 . Task loss averages Vg/max(1,Mg) over graph tasks. Neither selector inherits the tested finite-grid certificate.
k
Truth
Coverage
Available
Width
Action
Mean size
0
.3333
.5568
.6859
.1747
.6096
.6859
1
.3333
.7995
.8577
.8250
.6443
5.0026
2
.6859
.9786
1.0000
.9842
.7314
13.2315
3
.8577
.9831
1.0000
1.1113
.6909
26.0454
4
1.0000
.9831
1.0000
1.1965
.6525
44.2904
5
1.0000
.9831
1.0000
1.2382
.6315
64.4527
Appendix
Table 14: Fresh Oracle fixed- k retention sweep. The reference and all downstream quantities are held fixed while only the allowed semantic-disagreement count changes. The calibrated procedure selects k=4 in all 150 splits. Mean size counts retained DAGs and includes empty retained sets.
Policy
Truth
Coverage
Available
Width
Action
Act-and-harm
Data-PI
1.0000
.9831
1.0000
1.2633
.6234
.0000
Hard k=0
.3333
.5568
.6859
.1747
.6096
.0035
True-DAG k -set
1.0000
.9831
1.0000
1.1965
.6525
.0000
Size-matched
.7660
.9831
1.0000
1.2628
.6234
.0000
Appendix
Table 15: Fresh Oracle equal-network results. Within-network aggregation weights evaluation appearances. Width is conditional on interval availability. The true-DAG k -set uses the calibrated value k=4 .
Figure 9: Fresh Oracle retention under an adequate reference. Policies are Data-PI, hard k=0 , calibrated true-DAG disagreement k -set, and size-matched random retention. Panels show (a) exact truth retention, (b) effect coverage and interval availability, (c) width conditional on availability, and (d) action rate with joint act-and-harm annotated. Values weight evaluation appearances within each network and then weight the three networks equally. They are descriptive; no cross-network intervals are asserted. Member intervals use the 1.875th and 98.125th percentiles (96.25% nominal level). Retention calibration uses 1−δS=.925 , a different quantity.
Policy
Available draws
Available appearances
Data-PI
240
18,000
Hard k=0
165
12,388
True-DAG k -set
240
18,000
Size-matched
240
18,000
Appendix
Table 16: Oracle availability counts. Counts are pooled, whereas Table 15 reports equal-network rates. Each policy has 240 draws and 18,000 evaluation appearances.
Network
Policy
Truth
Coverage
Available
Width
Action
Mean size
Asia
Data-PI
1.0000
1.0000
1.0000
.7479
.7875
6.0
Hard
1.0000
.9875
1.0000
.2078
1.0000
1.0
k -set
1.0000
1.0000
1.0000
.7479
.7875
6.0
Size-matched
1.0000
1.0000
1.0000
.7479
.7875
6.0
Child
Data-PI
1.0000
.9500
1.0000
1.1671
.5750
24.0
Hard
.0000
.6750
1.0000
.1756
.7750
1.0
Appendix
Table 17: Fresh Oracle behavior by network. These entries average the 80 draw-level summaries equally within each network. They therefore use a different within-network weighting from Table 15 .
Policy
Closest
Available
Width
Coverage
Action
Act-and-harm
Data-PI
1.0000
1.0000
.8227
.6028
.7963
.0132
Hard k=0
.4563
.6637
.3269
.4361
.5896
.0123
Reference-disagreement k -set
.9965
1.0000
.8226
.6028
.7963
.0132
Size-matched
.8192
1.0000
.8219
.6012
.7963
.0132
Data-only
.4477
1.0000
.4728
.5181
.8618
.0132
Appendix
Table 18: Learned X6 downstream diagnostic. Closest denotes retention of the truth-informed closest stored member, not exact truth retention. The displayed k -set calibrates reference-member disagreement. Oracle results are not pooled with this population.
Policy
Exact
Closest
Available
Coverage
Width
Action
Mean size
k=0
.0428
.4563
.6637
.4361
.3269
.5896
.6898
k=1
.0428
.7074
.9162
.5343
.7917
.7387
4.1417
k=2
.0428
.9322
1.0000
.6028
.8169
.7963
10.7796
k=3
.0428
1.0000
1.0000
.6028
.8227
.7963
18.7037
k=4
.0428
1.0000
1.0000
.6028
.8227
.7963
25.8238
k=5
.0428
1.0000
1.0000
.6028
.8227
.7963
30.6512
Appendix
Table 19: Learned-reference fixed- k sweep. All downstream columns are pooled over the 4,320 evaluation appearances. Exact is exact true-DAG retention in this same appearance-weighted population; Closest is retention of the truth-informed closest stored member. Width conditions on availability. The calibrated row uses the split-specific reference-disagreement allowance rather than one fixed k .
Policy
Nonempty tasks
Available appearances
Mean size
Mean shrinkage
Zero-coded width
Data-PI
72
4320
34.1993
.0000
.8227
Hard k=0
48
2867
.6898
.9283
.2169
Reference-disagreement k -set
72
4320
18.4759
.1776
.8226
Size-matched
72
4320
18.4759
.1776
.8219
Data-only
72
4320
8.7106
.6207
.4728
Appendix
Table 20: X6 availability and pruning. Each policy has 72 usable tasks and 4,320 evaluation appearances. Nonempty-task counts and available-appearance counts have different units. Mean shrinkage is averaged per task or appearance rather than computed from aggregate sizes. Zero-coded width assigns zero when an interval is unavailable and is distinct from conditional width.
Construction
Records
Record failures
Planned coverage
Mean size
Median size
Exact MEC
80
0
.0375
44.5625
12
Skeleton-relaxed
74
6
.0500
280.3919
61
Bootstrap union
78
2
.0750
4291.4487
116
Appendix
Table 21: Reference construction on 80 planned tasks. Records denotes construction outputs produced by this ablation. Record failures are construction-stage failures and do not include the separate downstream enumeration-limit or valid-extension exclusions. Planned coverage uses all 80 tasks. Mean and median size condition on produced construction records.
τ
Candidates
Wrong recall
Correct queried
.50
0
.00
.000
.60
29
.32
.165
.70
57
.70
.278
.75
68
.80
.354
.80
80
.90
.443
.90
97
.98
.608
Appendix
Table 22: Instability-threshold sensitivity. Wrong recall concerns incorrectly oriented fitted arrows. Correct queried is the fraction of already-correct arrows selected.
Support
Candidates
Compelled
Reversible
False adjacency
Mean Directed-F1
.00
360
.511
.319
.169
.684
.25
287
.620
.380
.000
.681
.50
167
.689
.311
.000
.663
.75
52
.923
.077
.000
.639
Appendix
Table 23: Adjacency-support sensitivity. Compelled, reversible, and false-adjacency fractions describe the selected candidate pool. Higher compelled fraction does not imply higher recovery.
B
Mean candidates
Pooled Jaccard
Equal-network Jaccard
Mean support
Wrong committed
25
3.4
.737
.7933
.8827
9
50
3.2
.778
.6500
.9093
8
100
3.2
.778
.6500
.8982
7
250
3.2
1.000
1.0000
.9585
8
Appendix
Table 24: Five-network bootstrap diagnostic. Candidate means and equal-network Jaccard weight networks equally. Pooled Jaccard concerns the pooled edge-set comparison with B=250 . Semantic and matched-random recovery at these bootstrap counts are unavailable because the corresponding response records were not verified for this analysis.
Figure 10: Locator sensitivity measures different properties. (a) Wrong-arrow recall and already-correct-arrow query rate across instability thresholds. (b) Candidate composition across adjacency-support thresholds. (c) Mean Directed-F1 across those support thresholds. (d) Pooled and equal-network candidate-set Jaccard relative to B=250 . The sweeps use distinct populations. Candidate overlap does not establish a recovery benefit at larger bootstrap counts.
N
Paired n
PC Skel.-F1
G2G Skel.-F1
Δ Skel.-F1
PC Adj.-SHD
G2G Adj.-SHD
Δ Adj.-SHD
500
19
.7963
.8217
+.0254
12.632
12.263
−.368
2000
16
.8342
.8485
+.0142
12.375
11.625
−.750
Appendix
Table 25: Paired native Guess2Graph skeleton comparison. Absolute metrics are computed on the paired usable PC-family instances. Differences are Guess2Graph minus the shared PC baseline. This is a native skeleton-recovery comparison, not a matched-budget directional comparison.
Causal discovery seeks to uncover causal relations from data, typically represented as causal graphs, and is essential for predicting the effects of interventions. While expert knowledge is required to construct principled causal graphs, many statistical methods have been proposed to leverage observational data with varying formal guarantees. Causal Assumption-based Argumentation (ABA) is a framework that uses symbolic reasoning to ensure correspondence between input constraints and output graphs, while offering a principled way to combine data and expertise. We explore the use of large language models (LLMs) as imperfect experts for Causal ABA, eliciting semantic structural priors from variable names and descriptions and integrating them with conditional-independence evidence. Experiments on standard benchmarks and semantically grounded synthetic graphs demonstrate state-of-the-art performance, and we additionally introduce an evaluation protocol to mitigate memorisation bias when assessing LLMs for causal discovery.
Zihao Li, Fabrizio Russo
Department of Computing, Imperial College London, UK
Causal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it reliably remains an open question. Recent benchmarks show that even fine-tuned models plateau on simple causal graphs and degrade as complexity grows, but why they fail has not been established. We prove the failure is fundamental: supervised fine-tuning, direct preference optimization, and in-context learning all produce predictors that cannot distinguish between causal graphs generating similar observational data, and any attempt to do so requires the model's internal representations to grow unboundedly, violating the very conditions under which these methods work. We formalize this as a kernel obstruction theorem, establishing that the limitation is intrinsic to the learning paradigm, not any particular model or dataset. We propose Agentic Causal Bayesian Optimization (A-CBO), wherein a frozen language model serves as an interventional oracle answering targeted queries about intervention effects, while an external Bayesian loop concentrates beliefs over candidate graphs in logarithmically many rounds. Because the decision operates outside the space where the obstruction applies, A-CBO provably converges while the underlying model remains unchanged. On Corr2Cause, A-CBO matches fine-tuned baselines without any training. On Extended Corr2Cause, a new benchmark scaling to 24 variables with 18K test samples, A-CBO significantly outperforms both fine-tuning and preference optimization, with the advantage growing
Amartya Roy, Sonali Parbhoo
SIRE, IIT Delhi and Robert Bosch GmbH, India · Imperial College London
We present DMCD (DataMap Causal Discovery), a two-phase causal discovery framework that integrates LLM-based semantic drafting from variable metadata with statistical validation on observational data. In Phase I, a large language model proposes a sparse draft DAG, serving as a semantically informed prior over the space of possible causal structures. In Phase II, this draft is audited and refined via conditional independence testing, with detected discrepancies guiding targeted edge revisions. We evaluate our approach on three metadata-rich real-world benchmarks spanning industrial engineering, environmental monitoring, and IT systems analysis. Across these datasets, DMCD achieves competitive or leading performance against diverse causal discovery baselines, with particularly large gains in recall and F1 score. Probing and ablation experiments suggest that these improvements arise from semantic reasoning over metadata rather than memorization of benchmark graphs. Overall, our results demonstrate that combining semantic priors with principled statistical verification yields a high-performing and practically effective approach to causal structure learning.
Samarth KaPatel, Sofia Nikiforova, Giacinto Paolo Saggese +1