6.5% of the Neuro-Symbolic Literature Can Be Reproduced from Its Published Artifacts, a Six-Stage Audit Framework and First Instantiation
Organizations: Department of Computer Science, University of Maryland College Park, Maryland, USA
Abstract
We present a six-stage framework for auditing the reproducibility of scientific claims across a research literature within the computer science domain, and instantiate our framework for the neuro-symbolic AI (NSAI) subdomain. Instantiating the framework on the NSAI subdomain produced a multi-year audit. Stage one retrieved 5,497 records and removed 3,018 duplicates. Stage two screened the 2,479 unique records at title and abstract, identifying 1,365 self-identified NSAI records, then removed a further 61 at full text for off-topic, non-research, no-quantitative-evaluation, or inaccessible-full-text reasons. Stage three sought a verifiable public code artifact for each of the 1,304 eligible records and found none for 849, leaving 455 to enter the artifact inventory and bounded rerun of stages four and five. We fully or partially reproduced 85 studies, 6.52% of the eligible corpus and 18.68% of attempted reruns. We found that 321 attempted reruns were blocked by missing non- code artifacts and 42 by missing or unusable code repositories. These figures quantify a persistent reproducibility deficit that survives even nominal "code available" declarations, and signal the need for enforced, versioned, and permanently archived artifact bundles in future NSAI publications. We argue that empirical NSAI papers should be required at submission time to provide complete, versioned, and permanently archived artifact bundles.
Figures & tables
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Citation | Title | Description |
|---|---|---|
| Acharya et al. (2025) | ||
| Codebase | Neurosymbolic AI for Travel Demand Prediction: Integrating Decision Tree Rules into Neural Networks | This paper introduces a neuro-symbolic AI framework that blends decision tree rules with neural networks to predict travel demand. By combining interpretability with deep learning power, it achieves more accurate and explainable results for transportation planning and resource optimization. |
| Ishay et al. (2023) | ||
| Codebase | Leveraging Large Language Models to Generate Answer Set Programs | They use LLMs to work through logic puzzle solving in a step-by-step manner. |
| Murali et al. (2019) | ||
| Codebase | Composing Neural Learning and Symbolic Reasoning with an Application to Visual Discrimination | propose a compositional neurosymbolic framework that combines a neural network to detect objects and relationships with a symbolic learner that finds interpretable discriminators. |
Scientific Reproducibility
Scientific reproducibility studies whether research methods, data, and software can be inspected and rerun to support published findings.
See all 108 papers