We present a six-stage framework for auditing the reproducibility of scientific claims across a research literature within the computer science domain, and instantiate our framework for the neuro-symbolic AI (NSAI) subdomain. Instantiating the framework on the NSAI subdomain produced a multi-year audit. Stage one retrieved 5,497 records and removed 3,018 duplicates. Stage two screened the 2,479 unique records at title and abstract, identifying 1,365 self-identified NSAI records, then removed a further 61 at full text for off-topic, non-research, no-quantitative-evaluation, or inaccessible-full-text reasons. Stage three sought a verifiable public code artifact for each of the 1,304 eligible records and found none for 849, leaving 455 to enter the artifact inventory and bounded rerun of stages four and five. We fully or partially reproduced 85 studies, 6.52% of the eligible corpus and 18.68% of attempted reruns. We found that 321 attempted reruns were blocked by missing non- code artifacts and 42 by missing or unusable code repositories. These figures quantify a persistent reproducibility deficit that survives even nominal "code available" declarations, and signal the need for enforced, versioned, and permanently archived artifact bundles in future NSAI publications. We argue that empirical NSAI papers should be required at submission time to provide complete, versioned, and permanently archived artifact bundles.
Figures & tables
Fig. 1: Selection and eligibility flow for the NSAI audit. Late full-text eligibility exclusions are shown separately and are not counted as rerun outcomes. The dashed bottom box reports the number of successful reruns within the attempted set and is shown as an audit outcome.
Fig. 2: Rerun outcomes for the attempted same-artifact audit set ( n=455 ).
Fig. 3: Reproduction outcomes over time for the attempted NSAI papers. Stacked bars show, for each publication year, the number of papers that were successfully reproduced (full or partial) versus not reproduced under the study protocol, illustrating the growth of the field alongside a reproduction rate that shows no sustained upward trend.
Fig. 4: Reproduction outcomes by venue group for the attempted NSAI papers. Stacked bars show, for each venue family, the number of papers that were successfully reproduced (full or partial) versus not reproduced. Venues are ordered by number of attempted reruns.
Fig. 5: Rows indicate what was missing and columns indicate why . The plot contains non-exclusive, multi-label annotations across 363 attempted studies; a study may contribute to multiple cells, so cell counts do not sum to 363.
Fig. 6: Citation counts by attempted-rerun outcome/failure reason for NSAI papers with available citation data ( n=427 ) shown on a log scale. The red marker indicates the mean citation count for each bucket.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Citation
Title
Description
Acharya et al. (2025)
Codebase
Neurosymbolic AI for Travel Demand Prediction: Integrating Decision Tree Rules into Neural Networks
This paper introduces a neuro-symbolic AI framework that blends decision tree rules with neural networks to predict travel demand. By combining interpretability with deep learning power, it achieves more accurate and explainable results for transportation planning and resource optimization.
Ishay et al. (2023)
Codebase
Leveraging Large Language Models to Generate Answer Set Programs
They use LLMs to work through logic puzzle solving in a step-by-step manner.
Murali et al. (2019)
Codebase
Composing Neural Learning and Symbolic Reasoning with an Application to Visual Discrimination
propose a compositional neurosymbolic framework that combines a neural network to detect objects and relationships with a symbolic learner that finds interpretable discriminators.
AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.
Haokun Liu, Filbert Aurelian Tjiaranata, Chenhao Tan
Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles - the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date - demonstrate ARA's generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. ARA achieves ~61% accuracy on three benchmarks, and the highest accuracy reported on ReproBench (60.71% vs. 36.84%) and GoldStandardDB (61.68% vs. 43.56%), highlighting its potential to complement human review at scale and enabling next-generation peer review. Code and Data available: https://github.com/AndresLaverdeMarin/agentic_reproducibility_assessment.
Kevin Riehl, Andres L. Marin, Nikofors Zacharof +6
European Commission, Joint Research Centre · Ideas Forward
Scientific publication compresses a branching, iterative research process into a linear narrative, discarding the majority of what was discovered along the way. This compilation imposes two structural costs: a Storytelling Tax, where failed experiments, rejected hypotheses, and the branching exploration process are discarded to fit a linear narrative; and an Engineering Tax, where the gap between reviewer-sufficient prose and agent-sufficient specification leaves critical implementation details unwritten. Tolerable for human readers, these costs become critical when AI agents must understand, reproduce, and extend published work. We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package structured around four layers: scientific logic, executable code with full specifications, an exploration graph that preserves the failures compilation discards, and evidence grounding every claim in raw outputs. Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste. On PaperBench and RE-Bench, ARA raises question-answering accuracy from 72.4% to 93.7% and reproduction success from 57.4% to 64.4%. On RE-Bench's five open-ended extension tasks, preserved failure traces in ARA accelerate progress, but can also constrain a capable agent from stepping outside the prior-run box depending on the agent's capabilities. Our code is open-sourced at https://github.com/Orchestra-Research/Agent-Native-Research-Artifact.
Jiachen Liu, Jiaxin Pei, Jintao Huang +34
University of Michigan · Stanford University · Ohio State University +22