Large language models (LLMs) demonstrate remarkable breadth of knowledge, yet their ability to reason about computational processes remains poorly understood. Closing this gap matters for practitioners who rely on LLMs to guide algorithm selection and deployment. We address this limitation using causal discovery as a testbed and evaluate eight frontier LLMs against ground truth derived from algorithm executions. We find systematic, near-total failure across models. The predicted ranges are far wider than true confidence intervals yet still fail to contain the true algorithmic mean in most cases. Most models perform worse than random guessing. The best model's marginal improvement points to benchmark memorization rather than principled reasoning. We term this failure algorithmic blindness and argue it reflects a fundamental gap between declarative knowledge about algorithms and calibrated procedural prediction.
Figures & tables
Figure 1: Comparison of LLM estimates and algorithmic ground truth revealing algorithmic blindness. The model is prompted to predict performance ranges for the PC algorithm on the Sachs dataset and the predicted ranges are wide yet fail to contain the true values.
Table 2
Algorithm
Benchmark Cov.
Synthetic Cov.
Difference
FCI
25.0%
43.8%
+18.8%
NOTEARS
44.4%
68.8%
+24.3%
PC
27.8%
43.8%
+16.0%
LiNGAM
47.2%
31.2%
−16.0%
Table 3: Claude’s calibrated coverage by algorithm and dataset type.
Method
Coverage (%)
Mean Score
Random Baseline
36.5
0.409
Heuristic Baseline
32.7
0.356
Claude
39.4
0.442
GPT-5
15.4
0.217
DeepSeek-Think
14.9
0.174
DeepSeek
14.4
0.198
Table 4: Calibrated Coverage: LLMs versus Baselines
Figure 5Figure 6
Dataset
Type
Mean Dist.
Agree.%
Asia
Benchmark
2.20
44.2
Cancer
Benchmark
1.88
34.8
Earthquake
Benchmark
2.01
43.5
Survey
Benchmark
2.13
43.3
Sachs
Benchmark
4.55
47.8
Child
Benchmark
8.62
46.2
Table 6: Cross-Model Agreement by Dataset (Mean Pairwise Distance and Agreement%)
Figure 5: (a) Cross-model pairwise distance on benchmark versus synthetic datasets. (b) Mean cross-model pairwise distance as a function of synthetic network size.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Three prompt formulations used across all experimental conditions. Formulation 1 elicits direct numerical estimates; Formulation 2 guides explicit step-by-step reasoning about algorithm assumptions; Formulation 3 frames the task as confidence interval estimation over repeated runs. All three require identical output format to enable CV% comparison.
Metric
Claude
DeepSeek
DeepSeek-Think
Gemini 3
GPT-5
LLaMA
Qwen
Qwen-Think
F1
17.1%
18.3%
14.0%
13.9%
13.9%
7.5%
5.3%
23.0%
Precision
15.4%
19.6%
15.3%
13.9%
11.9%
6.4%
4.1%
22.7%
Recall
18.5%
18.8%
14.0%
15.4%
15.8%
8.3%
6.2%
23.2%
SHD
19.4%
41.5%
36.4%
23.0%
26.5%
37.1%
30.4%
35.1%
Overall
17.6%
24.6%
19.9%
16.5%
17.0%
14.8%
11.5%
26.0%
Appendix
Table 7: Prompt Robustness (CV%) by Metric and Model
Asia (Benchmark, 8 nodes)
Synthetic-12 (Synthetic, 12 nodes)
Algorithm
Metric
Algo Mean
LLM Range
Coverage
Algorithm
Metric
Algo Mean
LLM Range
Coverage
PC
Precision
0.474
[0.703, 0.874]
12.5%
PC
Precision
0.470
[0.688, 0.860]
12.5%
PC
Recall
0.777
[0.626, 0.811]
87.5%
PC
Recall
0.723
[0.613, 0.797]
75.0%
PC
F1
0.588
[0.664, 0.831]
0.0%
PC
F1
0.569
[0.650, 0.815]
0.0%
PC
SHD
15.0
[2.1, 6.2]
0.0%
PC
SHD
32.8
[4.9, 13.6]
0.0%
FCI
Precision
0.474
[0.666, 0.853]
12.5%
FCI
Precision
0.472
[0.616, 0.805]
12.5%
Appendix
Table 8: Calibrated Coverage by Algorithm and Metric for Asia and Synthetic-12 datasets
Dataset
Type
Nodes
Coverage (%)
Asia
Benchmark
8
23.4
Cancer
Benchmark
5
21.9
Synthetic-12
Synthetic
12
20.3
Alarm
Benchmark
37
18.8
Insurance
Benchmark
27
18.8
Survey
Benchmark
6
18.0
Appendix
Table 9: Calibrated Coverage by Dataset (Aggregated Across All Algorithms and Metrics)
Algorithm
Avg Synthetic Boost
Range Variation
NOTEARS
+1.7%
56.9%
FCI
−0.5%
34.7%
PC
−2.0%
24.3%
LiNGAM
−23.2%
44.4%
Appendix
Table 10: Average Synthetic Coverage Boost by Algorithm for all models.
Algorithm
Precision
Recall
F1
SHD
PC
0.335
0.868
0.427
130.36
FCI
0.343
0.839
0.426
122.67
LiNGAM
0.304
0.405
0.301
101.00
NOTEARS
0.341
0.248
0.299
36.77
Appendix
Table 11: Algorithmic Ground Truth: Mean Performance Across 13 Datasets