cs.CRNov 16, 2026

ReconPotBench: Evaluating LLM Honeypots in Multi-Round Linux Reconnaissance

Authors: Qianli Wang, Junchao Wu, Binkai Ou, Lidia S. Chao, Derek F. Wong

Organizations: NLP2CT Lab, Faculty of Information Science and Computing, University of Macau · School of Computer Science and Technology, Shandong University · BoardWare Information System Limited

Abstract

Large language models (LLMs) are increasingly used to build interactive honeypots, yet existing evaluations mainly assess isolated command outputs and cannot determine whether a system remains believable when later commands depend on earlier results. We present ReconPotBench, a multi-round benchmark for Linux reconnaissance honeypots. It contains 1{,}032 three-round reconnaissance interactions across eight reconnaissance categories, with reference outputs collected from a real Linux environment. Under a unified testing protocol, we compare Cowrie, open-source LLMs, and closed-source LLMs along three dimensions: round-level authenticity, cross-round contextual consistency, and hacker-oriented reconnaissance value. Results show that multi-round evaluation exposes failures hidden by single-round testing, as authenticity declines with interaction depth and both open-source and closed-source LLMs struggle to maintain coherent state across rounds. LLM-based honeypots nevertheless outperform Cowrie in authenticity and reconnaissance value, while Cowrie remains more consistent because of its rigid templated behavior. Category-level analysis further shows that tasks requiring systems to preserve network relationships across rounds are harder than tasks centered on explicit entities such as ports. ReconPotBench enables reproducible evaluation of LLM honeypots across authenticity, entity consistency, and hacker-perspective utility in dependent multi-round interactions. The dataset is available at: https://github.com/qianli-wang/ReconPotBench

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots

    May 28, 2026Mark Vero, Fabian Kaczmarczyck, Ivan Petrov +6LLMs for Cybersecurity