When Should Semantic Priors Enter Partially-Identified Causal Discovery?
Organizations: Stanford University · FAST-NUCES Lahore, Pakistan · University of Virginia Charlottesville, USA · VERSES Technologies Inc. ·
Abstract
Observational causal discovery often leaves causal di- rections unresolved or unstable. Large language mod- els (LLMs) can provide semantic information to resolve these directions, but incorrect proposals can exclude the true causal graph and invalidate downstream inference. We study three separate decisions: where to allocate semantic queries, which proposals to admit, and which causal graphs to retain. Our framework distinguishes population non-identifiability from finite-sample insta- bility, evaluates semantic orientation against statistical and random controls, and examines calibrated admis- sion and error-tolerant DAG retention. Experiments across benchmark networks show that semantic infor- mation improves recovery over coin-flip orientations, but targeting unstable edges has no established ad- vantage over matched-random allocation. The tested admission certificate cannot support nontrivial admis- sion with the available calibration tasks. Moreover, downstream retention depends on reference adequacy: learned references contain the true DAG in only three of 80 planned tasks. These findings separate the value of semantic information from the reliability of its alloca- tion, admission, and retention, and identify calibration size and reference adequacy as key limitations.
Figures & tables
| PC | BOSS | |||
|---|---|---|---|---|
| Policy | 500 | 2000 | 500 | 2000 |
| Base | .4011 | .4962 | .6256 | .6500 |
| F1-only | .6177 | .6831 | .7744 | .8414 |
| F1+rand | .6176 | .7179 | .7831 | .8450 |
| F1+F2-data | .6206 | .6837 | .7662 | .8405 |
| Bootstrap–LLM | .6215 | .6867 | .7685 | .8488 |
| Target risk | .10 | .20 | .30 |
|---|---|---|---|
| Hoeffding, zero loss | 205 | 52 | 23 |
| Exact, zero loss events | 39 | 19 | 12 |
| F1 available | 10 graph tasks | ||
| F2 available | 9 graph tasks | ||
| Hoeffding selector | Complete abstention | ||
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Network | Nodes | Edges | Recovery panel |
|---|---|---|---|
| Asia | 8 | 8 | Primary |
| Sachs | 11 | 17 | Primary |
| Child | 20 | 25 | Primary |
| Insurance | 27 | 52 | Primary |
| Alarm | 37 | 46 | Primary |
| Barley | 48 | 84 | Additional |
| Policy | PC-500 | PC-2000 | BOSS-500 | BOSS-2000 |
|---|---|---|---|---|
| Base | .4011 / 23.44 | .4962 / 20.75 | .6256 / 15.00 | .6500 / 14.88 |
| F1-only | .6177 / 17.60 | .6831 / 15.50 | .7744 / 10.08 | .8414 / 8.24 |
| F1+rand | .6176 / 17.68 | .7179 / 14.75 | .7831 / 10.04 | .8450 / 8.20 |
| F1+F2-data | .6206 / 17.60 | .6837 / 15.55 | .7662 / 10.32 | .8405 / 7.92 |
| Bootstrap–LLM | .6215 / 17.88 | .6867 / 15.75 | .7685 / 10.76 | .8488 / 8.24 |
| Propagation | .6184 / 17.36 | .7064 / 14.95 | .7811 / 9.96 | .8462 / 8.12 |
| Baseline | Initial | Intermediate | Final |
|---|---|---|---|
| Bootstrap–LLM | 285 eligible | 141 agree | 132 admitted |
| Propagation | 285 selected | 260 responses | 25 missing |
| Contrast | Setting | Pooled F1 | Equal-network F1 | 95% interval | Pooled SHD |
| F2 minus random | PC-500 | +.01672 | +.01672 | ||
| PC-2000 | +.25 | ||||
| BOSS-500 | .00 | ||||
| BOSS-2000 | +.01596 | +.01596 | |||
| Semantic minus data | PC-500 | +.01375 | +.01375 | ||
| PC-2000 | +.01940 | +.02406 |
| Contrast | Estimate | 95% interval |
|---|---|---|
| Semantic minus coin | +.08263 | |
| Coin minus Base | +.10363 | |
| Semantic minus Base | +.18626 |
| Network | PC-500 | PC-2000 | BOSS-500 | BOSS-2000 |
|---|---|---|---|---|
| F1+F2 minus F1+rand | ||||
| Asia | +.0214 | .0000 | .0000 | |
| Sachs | .0000 | +.0125 | ||
| Child | +.0160 | .0000 | ||
| Insurance | +.0138 | +.0304 | ||
| Alarm | +.0801 | +.0089 | +.0369 | |
| Cap | PC-500 | PC-2000 | BOSS-500 | BOSS-2000 |
|---|---|---|---|---|
| 0 | .0000 / .00 | .0000 / .00 | .0000 / .00 | .0000 / .00 |
| 10 | .0000 / .00 | .0000 / .00 | .0000 / .00 | .0000 / .00 |
| 20 | / +.04 | .0000 / .00 | .0000 / .00 | .0000 / .00 |
| 40 | / +.08 | / +.20 | / +.12 | .0000 / .00 |
| 80 | +.0054 / | / +.15 | / +.12 | +.0056 / |
| Full | +.0167 / | / +.25 | / .00 | +.0160 / |
| Contrast | Population | Networks | Effect | 95% interval |
|---|---|---|---|---|
| F2 minus random | Primary | 5 | +.00474 | |
| Additional | 3 | |||
| Combined sensitivity | 8 | +.00257 | ||
| Semantic minus data | Primary | 5 | +.01718 | |
| Additional | 3 | +.00418 | ||
| Combined sensitivity | 8 | +.01230 |
| Condition | Raw correct | Valid-only correct | UNKNOWN | Contradiction |
|---|---|---|---|---|
| Named | 1143/1280 | 1143/1230 | 146/1760 | 32/774 |
| No-Benchmark | 1114/1280 | 1114/1231 | 132/1760 | 50/783 |
| Anonymous | 1052/1280 | 1052/1214 | 139/1760 | 79/778 |
| Shuffled | 705/1280 | 705/963 | 538/1760 | 95/522 |
| Condition | Representation | Raw correct | Valid-only correct | UNKNOWN | Contradiction |
|---|---|---|---|---|---|
| Original | Canonical | 305/380 | 305/355 | 25/380 | 20/175 |
| Original | Permuted | 316/380 | 316/360 | 20/380 | 24/180 |
| Anonymous | Canonical | 245/380 | 245/335 | 45/380 | 35/150 |
| Anonymous | Permuted | 242/380 | 242/325 | 55/380 | 31/150 |
| Shuffled | Canonical | 141/380 | 141/256 | 124/380 | 20/110 |
| Shuffled | Permuted | 124/380 | 124/263 | 117/380 | 21/119 |
| Policy | Pool | Admission | Task loss |
|---|---|---|---|
| Add-one | F1 | 1.000 | .162 |
| Add-one | F2 | .805 | .175 |
| Agreement | F1 | .944 | .170 |
| Agreement | F2 | .965 | .198 |
| Truth | Coverage | Available | Width | Action | Mean size | |
|---|---|---|---|---|---|---|
| 0 | .3333 | .5568 | .6859 | .1747 | .6096 | .6859 |
| 1 | .3333 | .7995 | .8577 | .8250 | .6443 | 5.0026 |
| 2 | .6859 | .9786 | 1.0000 | .9842 | .7314 | 13.2315 |
| 3 | .8577 | .9831 | 1.0000 | 1.1113 | .6909 | 26.0454 |
| 4 | 1.0000 | .9831 | 1.0000 | 1.1965 | .6525 | 44.2904 |
| 5 | 1.0000 | .9831 | 1.0000 | 1.2382 | .6315 | 64.4527 |
| Policy | Truth | Coverage | Available | Width | Action | Act-and-harm |
|---|---|---|---|---|---|---|
| Data-PI | 1.0000 | .9831 | 1.0000 | 1.2633 | .6234 | .0000 |
| Hard | .3333 | .5568 | .6859 | .1747 | .6096 | .0035 |
| True-DAG -set | 1.0000 | .9831 | 1.0000 | 1.1965 | .6525 | .0000 |
| Size-matched | .7660 | .9831 | 1.0000 | 1.2628 | .6234 | .0000 |
| Policy | Available draws | Available appearances |
|---|---|---|
| Data-PI | 240 | 18,000 |
| Hard | 165 | 12,388 |
| True-DAG -set | 240 | 18,000 |
| Size-matched | 240 | 18,000 |
| Network | Policy | Truth | Coverage | Available | Width | Action | Mean size |
|---|---|---|---|---|---|---|---|
| Asia | Data-PI | 1.0000 | 1.0000 | 1.0000 | .7479 | .7875 | 6.0 |
| Hard | 1.0000 | .9875 | 1.0000 | .2078 | 1.0000 | 1.0 | |
| -set | 1.0000 | 1.0000 | 1.0000 | .7479 | .7875 | 6.0 | |
| Size-matched | 1.0000 | 1.0000 | 1.0000 | .7479 | .7875 | 6.0 | |
| Child | Data-PI | 1.0000 | .9500 | 1.0000 | 1.1671 | .5750 | 24.0 |
| Hard | .0000 | .6750 | 1.0000 | .1756 | .7750 | 1.0 |
| Policy | Closest | Available | Width | Coverage | Action | Act-and-harm |
|---|---|---|---|---|---|---|
| Data-PI | 1.0000 | 1.0000 | .8227 | .6028 | .7963 | .0132 |
| Hard | .4563 | .6637 | .3269 | .4361 | .5896 | .0123 |
| Reference-disagreement -set | .9965 | 1.0000 | .8226 | .6028 | .7963 | .0132 |
| Size-matched | .8192 | 1.0000 | .8219 | .6012 | .7963 | .0132 |
| Data-only | .4477 | 1.0000 | .4728 | .5181 | .8618 | .0132 |
| Policy | Exact | Closest | Available | Coverage | Width | Action | Mean size |
|---|---|---|---|---|---|---|---|
| .0428 | .4563 | .6637 | .4361 | .3269 | .5896 | .6898 | |
| .0428 | .7074 | .9162 | .5343 | .7917 | .7387 | 4.1417 | |
| .0428 | .9322 | 1.0000 | .6028 | .8169 | .7963 | 10.7796 | |
| .0428 | 1.0000 | 1.0000 | .6028 | .8227 | .7963 | 18.7037 | |
| .0428 | 1.0000 | 1.0000 | .6028 | .8227 | .7963 | 25.8238 | |
| .0428 | 1.0000 | 1.0000 | .6028 | .8227 | .7963 | 30.6512 |
| Policy | Nonempty tasks | Available appearances | Mean size | Mean shrinkage | Zero-coded width |
|---|---|---|---|---|---|
| Data-PI | 72 | 4320 | 34.1993 | .0000 | .8227 |
| Hard | 48 | 2867 | .6898 | .9283 | .2169 |
| Reference-disagreement -set | 72 | 4320 | 18.4759 | .1776 | .8226 |
| Size-matched | 72 | 4320 | 18.4759 | .1776 | .8219 |
| Data-only | 72 | 4320 | 8.7106 | .6207 | .4728 |
| Construction | Records | Record failures | Planned coverage | Mean size | Median size |
|---|---|---|---|---|---|
| Exact MEC | 80 | 0 | .0375 | 44.5625 | 12 |
| Skeleton-relaxed | 74 | 6 | .0500 | 280.3919 | 61 |
| Bootstrap union | 78 | 2 | .0750 | 4291.4487 | 116 |
| Candidates | Wrong recall | Correct queried | |
|---|---|---|---|
| .50 | 0 | .00 | .000 |
| .60 | 29 | .32 | .165 |
| .70 | 57 | .70 | .278 |
| .75 | 68 | .80 | .354 |
| .80 | 80 | .90 | .443 |
| .90 | 97 | .98 | .608 |
| Support | Candidates | Compelled | Reversible | False adjacency | Mean Directed-F1 |
|---|---|---|---|---|---|
| .00 | 360 | .511 | .319 | .169 | .684 |
| .25 | 287 | .620 | .380 | .000 | .681 |
| .50 | 167 | .689 | .311 | .000 | .663 |
| .75 | 52 | .923 | .077 | .000 | .639 |
| Mean candidates | Pooled Jaccard | Equal-network Jaccard | Mean support | Wrong committed | |
|---|---|---|---|---|---|
| 25 | 3.4 | .737 | .7933 | .8827 | 9 |
| 50 | 3.2 | .778 | .6500 | .9093 | 8 |
| 100 | 3.2 | .778 | .6500 | .8982 | 7 |
| 250 | 3.2 | 1.000 | 1.0000 | .9585 | 8 |
| Paired | PC Skel.-F1 | G2G Skel.-F1 | Skel.-F1 | PC Adj.-SHD | G2G Adj.-SHD | Adj.-SHD | |
|---|---|---|---|---|---|---|---|
| 500 | 19 | .7963 | .8217 | +.0254 | 12.632 | 12.263 | |
| 2000 | 16 | .8342 | .8485 | +.0142 | 12.375 | 11.625 |
Causal Discovery
Causal discovery infers cause-and-effect structures from observational and interventional data, distinguishing causal relationships from correlation.
See all 239 papers