Do LLMs Understand Network Traces? A Comparative Study of Reasoning over Raw and Filtered Traffic Data
Organizations: Network Engineering & Security Group - CITIC School of Computer Science and Telecommunication Engineering University of Granada Granada, Spain
Abstract
The effective delivery and monitoring of ICT services rely heavily on network traffic analysis. However, interpreting complex packet captures remains a challenging task that requires substantial technical expertise. Large language models (LLMs) offer promising opportunities to automate network traffic in- vestigation, yet their reliability in packet-level field extraction and flow-level sequence reconstruction remains an open research question. This study proposes a reproducible, question-answering- based evaluation methodology to assess the capabilities of LLMs in network trace analysis. Using real-world benign traffic from the CICIDS2017 dataset, we evaluate multiple models against a human-generated ground truth, comparing their performance on raw textual traffic dumps and protocol-filtered representations. Model responses are assessed through an automated LLM- as-a-judge pipeline, validated against human evaluations. The results demonstrate that protocol filtering significantly reduces prompt overhead while improving answer accuracy by reducing incorrect responses. These findings highlight the importance of protocol-aware traffic filtering as an effective optimization for improving the efficiency and reliability of LLM-assisted network management.
Figures & tables
| Model | Parameters | Context | Quant. | Role |
|---|---|---|---|---|
| Llama 3.1 | 8B | 131,072 | Q4_K_M | Answer generation |
| Llama 3.1 | 70.6B | 131,072 | Q4_K_M | Answer generation |
| Qwen 2.5 | 7.6B | 32,768 | Q4_K_M | Answer generation |
| Qwen 2.5 | 72.7B | 32,768 | Q4_K_M | Answer generation |
| Llama 3.3 | 70.6B | 131,072 | Q4_K_M | LLM-as-a-judge |
| Packets | ||||||
|---|---|---|---|---|---|---|
| Trace | Traffic | Default | ARP | IP | TCP | UDP |
| T1 | Host initialisation (neighbour discovery, address resolution, connectivity checks) | 23 | 7 | 6 | 0 | 0 |
| T2 | Connectivity tests, DNS, IP fragmentation, failed TCP connections | 29 | 0 | 29 | 7 | 6 |
| T3 | Nmap-based TCP SYN and UDP scan, ICMP from closed ports | 32 | 4 | 27 | 13 | 8 |
| T4 | Host discovery and port scanning using ARP, UDP, TCP, and ICMP, including TCP three-way handshake completion | 34 | 4 | 30 | 16 | 8 |
| Model | Portion (%) | View | L2 (ARP) | L3 (IP) | L4 (TCP/UDP) | Quantitative overview | Qualitative overview | ||||||||
| C | P | I | C | P | I | C | P | I | C | P | I | ||||
| Qwen 2.5 7B | 50% | Def. | 60 | 13.3 | 26.7 | 5 | 45 | 50 | 51.7 | 6.7 | 41.7 | 43.2 | 15.8 | 41.1 | |
| Fil. | 53.3 | 15 | 25 | 50 | 36.7 | 32.6 | |||||||||
| 75% | Def. | 46.7 | 13.3 | 40 | 5 | 45 | 50 | 21.6 | 21.7 | 56.7 | 22.1 | 25.3 | 52.6 | ||
| Fil. | 26.6 | 15 | 10 | 23.3 | 41.7 | 25.3 | 32.6 | ||||||||
| 100% | Def. | 60 | 6.7 | 33.3 | 5 | 40 | 55 | 20 | 18.3 | 61.7 | 23.2 | 21.1 | 55.8 | ||
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,527 papers