Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers
Authors: David Heye, Karl Kindermann, Robin Decker, Johannes Lohmöller, Anastasiia Belova, Sandra Geisler, Klaus Wehrle, Jan Pennekamp
Organizations: Communication and Distributed Systems, RWTH Aachen University, Germany · Data Stream Management and Analysis, RWTH Aachen University, Germany
Artifact Evaluation (AE) is essential for ensuring the transparency and reliability of research, closing the gap between exploratory work and real-world deployment is particularly important in cybersecurity, particularly in IoT and CPSs, where large-scale, heterogeneous, and privacy-sensitive data meet safety-critical actuation. Yet, manual reproducibility checks are time-consuming and do not scale with growing submission volumes. In this work, we demonstrate that Large Language Models (LLMs) can provide powerful support for AE tasks: (i) text-based reproducibility rating, (ii) autonomous sandboxed execution environment preparation, and (iii) assessment of methodological pitfalls. Our reproducibility-assessment toolkit yields an accuracy of over 72% and autonomously sets up execution environments for 28% of runnable cybersecurity artifacts. Our automated pitfall assessment detects seven prevalent pitfalls with high accuracy (F1 > 92%). Hence, the toolkit significantly reduces reviewer effort and, when integrated into established AE processes, could incentivize authors to submit higher-quality and more reproducible artifacts. IoT, CPS, and cybersecurity conferences and workshops may integrate the toolkit into their peer-review processes to support reviewers' decisions on awarding artifact badges, improving the overall sustainability of the process.
Figures & tables
Fig. 1: The three pipeline stages require different inputs, and each stage utilizes an Large Language Model. Further, they can be used in combination as desired. Any available results can then be fed into the Artifact Evaluation ( Complement stage).
Fig. 2: Mapping of concept vectors with a known concept “reproducible results.” When measuring a new vector “artifacts available,” it is mapped via a projection to the original vector to compute a score.
Fig. 3: The Large Language Model agent gets access to the paper, the relevant source code and data, as well as a Readme (if available). It then generates commands to execute the code and runs them in a terminal. The outputs are sent back to the agent to determine the next steps.
Total 126
Olszewski-Study
runs
¬ runs
Pipeline
runs
7.14%
8.73%
15.87%
¬ runs
19.05%
65.08%
84.13%
26.19%
73.81%
TABLE I: Comparison of the output of the reproducibility pipeline with the Olszewski-Study . The pipeline correctly classifies almost three-quarters of the examined submissions, providing execution environments for more than 27% of all submissions marked as runnable in the ground-truth.
Total 130
Olszewski-Study
runs
¬ runs
Rate
runs
40.77%
54.62%
95.38%
¬ runs
2.31%
2.31%
4.62%
43.08%
56.92%
TABLE II: Comparison of the output of the Rate stage with the Olszewski-Study . The approach correctly classifies almost all submissions marked as runnable in the ground-truth.
Total 311
Olszewski-Study
runs
¬ runs
Prepare
runs
7.40%
14.79%
22.19%
¬ runs
18.97%
58.84%
77.81%
26.37%
73.63%
TABLE III: Comparison of the output of the Prepare stage with the Olszewski-Study . The agent automatically sets up ready-to-use execution environments for almost 29% of all submissions marked as runnable in the ground-truth.