ReconPotBench: Evaluating LLM Honeypots in Multi-Round Linux Reconnaissance
Organizations: NLP2CT Lab, Faculty of Information Science and Computing, University of Macau · School of Computer Science and Technology, Shandong University · BoardWare Information System Limited
Abstract
Large language models (LLMs) are increasingly used to build interactive honeypots, yet existing evaluations mainly assess isolated command outputs and cannot determine whether a system remains believable when later commands depend on earlier results. We present ReconPotBench, a multi-round benchmark for Linux reconnaissance honeypots. It contains 1{,}032 three-round reconnaissance interactions across eight reconnaissance categories, with reference outputs collected from a real Linux environment. Under a unified testing protocol, we compare Cowrie, open-source LLMs, and closed-source LLMs along three dimensions: round-level authenticity, cross-round contextual consistency, and hacker-oriented reconnaissance value. Results show that multi-round evaluation exposes failures hidden by single-round testing, as authenticity declines with interaction depth and both open-source and closed-source LLMs struggle to maintain coherent state across rounds. LLM-based honeypots nevertheless outperform Cowrie in authenticity and reconnaissance value, while Cowrie remains more consistent because of its rigid templated behavior. Category-level analysis further shows that tasks requiring systems to preserve network relationships across rounds are harder than tasks centered on explicit entities such as ports. ReconPotBench enables reproducible evaluation of LLM honeypots across authenticity, entity consistency, and hacker-perspective utility in dependent multi-round interactions. The dataset is available at: https://github.com/qianli-wang/ReconPotBench
Figures & tables
| Metric | Matched / Total | Agreement |
|---|---|---|
| Authenticity | 1,134 / 1,248 | 90.87% |
| Consistency | 405 / 416 | 97.36% |
| Utility | 380 / 416 | 91.35% |
| System | Authenticity Score | Consistency Score | Hacker Utility Score | Rank | |||
| R1 | R2 | R3 | Avg. | ||||
| Traditional Honeypot | |||||||
| Cowrie | 0.21 | 0.15 | 0.15 | 0.17 | 0.45 | 0.66 | 17 |
| Closed-source LLM | |||||||
| Sonnet-4.6 | 0.36 | 0.33 | 0.19 | 0.29 | 0.11 | 3.24 | 1 |
| Doubao-1.5 | 0.26 | 0.21 | 0.09 | 0.19 | 0.08 | 2.70 | 13 |
| Category | R1 | R2 | R3 | Auth. Avg. | Consistency | Utility Avg. |
|---|---|---|---|---|---|---|
| Network Interfaces & IP | 0.32 | 0.26 | 0.16 | 0.25 | 0.04 | 3.17 |
| Open Ports & Listening Services | 0.49 | 0.36 | 0.22 | 0.36 | 0.18 | 3.40 |
| Port Connection States | 0.26 | 0.28 | 0.35 | 0.30 | 0.20 | 3.64 |
| Routing Table | 0.14 | 0.34 | 0.05 | 0.18 | 0.03 | 2.93 |
| Service Versions & Components | 0.60 | 0.32 | 0.16 | 0.36 | 0.08 | 3.35 |
| DNS Configuration | 0.31 | 0.19 | 0.12 | 0.21 | 0.02 | 2.98 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | Value |
|---|---|
| Inference framework | transformers + PyTorch |
| Sampling strategy | do_sample=False |
| Maximum generation length | max_new_tokens=1024 |
| GPU | NVIDIA A100-PCIE-40GB |
| CPU | 10 Xeon vCPUs |
| Operating system | Ubuntu 22.04 |
| ID | Round 1 Network-layer | Round 2 Software-layer | Round 3 Hardware-layer |
|---|---|---|---|
| 1 | Network Interfaces & IP | Interface count, address type, and bound services | interface-related hardware |
| 2 | Open Ports & Listening Services | Port-to-service mapping and version clues | Service-specific hardware needs |
| 3 | Port Connection States | Client/server software behind active connections | Load-related hardware bottlenecks |
| 4 | Routing Table | Gateway/subnet clues for routing or firewall software | Routing and NIC hardware support |
| 5 | Service Versions & Components | Package, component, and vulnerability-related software clues | Deployment-related hardware support |
| 6 | DNS Configuration | DNS software stack | Local network role |
LLMs for Cybersecurity
Large language models for cybersecurity support threat analysis, vulnerability research, security operations, and cyber defense.
See all 282 papers