What's on My Network? Using Large Language Models to Identify Real-World IoT Devices at Scale
Authors: Rameen Mahmood, Tousif Ahmed, Sai Teja Peddinti, Danny Yuxing Huang
Abstract
The growth of IoT devices in shared environments has outpaced our ability to identify them, posing urgent risks to privacy, safety, and accountability. This challenge is especially pronounced in open-world environments, where network traffic metadata is often sparse, noisy, or adversarial. To address this problem, we introduce a semantic inference pipeline that reframes device identification as a language modeling task over real-world network metadata. As this approach depends on reliable supervision, we first construct high-fidelity vendor labels for the IoT Inspector dataset, the largest real-world corpus of its kind, using an ensemble of large language models guided by mutual-information and entropy-based stability scores. We then instruction-tune a quantized LLaMA 3.1 8B model on this dataset using curriculum learning to support generalization under sparsity and long-tail vendor distributions. Our model achieves 98.69% top-1 and 90.73% macro accuracy across 2,015 vendors, while remaining robust to missing fields, protocol drift, and adversarial manipulation. We also evaluate the model on an independent IoT testbed dataset, assess explanation quality, and conduct adversarial tests to probe robustness under spoofed and obfuscated input. These results position instruction-tuned LLMs as a scalable, interpretable foundation for trustworthy device identification at scale.
Internet of Things (IoT) systems are inherently vulnerable due to constrained hardware, outdated firmware, and insecure default configurations, creating a need for scalable and adaptive security testing approaches. While recent adoptions of Large Language Model (LLM) agents have demonstrated promise in penetration testing and Capture-the-Flag (CTF) environments, their application to IoT specific vulnerabilities remains unexplored. This paper presents an autonomous multi-agent framework, referred to as Vulnerability EXploitation using AI Agents (VEXAIoT), for vulnerability discovery and exploitation in IoT environments using LLM-based reasoning and offensive security tools. The framework combines a vulnerability detection agent and an attack execution agent to perform reconnaissance, plan attack sequences, and execute exploits against vulnerable IoT services. The system is evaluated in IoTGoat and Metasploitable environments across ten attack scenarios mapped to OWASP IoT vulnerabilities. Experimental results show attack success rate of up to 100% with low token overhead and average execution times under two minutes for most attacks. Across 260 attack executions, VEXAIoT achieves a 95.0% overall success rate, including 94.5% success in IoTGoat and 96.7% success in Metasploitable2. These results demonstrate the potential for LLM-driven agents to automate IoT vulnerability assessment and offensive security workflows in controlled environments
The rapid proliferation of Internet of Things (IoT) devices has created an urgent demand for adaptive, resource-efficient Intrusion Detection Systems (IDS) capable of handling dynamic and evolving cyber threats. This paper investigates AOC-IDS, a state-of-the-art autonomous online IDS published at IEEE INFOCOM 2024, which employs an Autoencoder (AE) with Cluster Repelling Contrastive (CRC) loss and an autonomous Gaussian-based decision module. We first successfully replicate AOC-IDS on the UNSW-NB15 benchmark, achieving 89.39% accuracy in close agreement with the published 89.19%. We then identify four key limitations: class imbalance, unreliable pseudo-label generation, limited generalization, and computational overhead for IoT deployment, and propose targeted improvements for each. Our XGBoost-BalSamp method achieves 95.45% accuracy on UNSW-NB15, a gain of 6.26% over the baseline. Our combined deep learning approach (PseudoFilter, MixupAug, and LiteAE) achieves a best-run accuracy of 90.88% (F1: 91.45%), surpassing the base paper while reducing model parameters by 55%.These results demonstrate that targeted improvements to AOC-IDS yield consistent accuracy gains while improving practical deployability on IoT edge devices.
Some low-cost Internet of Things (IoT) sensor deployments lack device-level source authentication, leaving them vulnerable to impersonation or injected sensor readings. We present a lightweight approach to sensor impersonation detection in a small proof-of-concept study. We formulate detection as a sequence-prediction problem. A model with three LSTM layers and two fully connected layers is trained only on univariate temperature readings from a genuine sensor, and a window of readings is flagged when its mean absolute prediction error exceeds the mean genuine error by more than six standard deviations. The model is converted to TensorFlow Lite in three variants, non-quantized (554 KB), 16-bit weight quantized (298.5 KB), and 8-bit weight quantized (185 KB), targeting an Arduino Nano 33 BLE aggregator. The variants were evaluated with the TensorFlow Lite interpreter on the local server, since on-device execution of LSTM models was not yet supported by TensorFlow Lite for Microcontrollers at the time of the study. On a controlled testbed with the impostor sensor placed in a hotter outdoor location, the three variants reached detection accuracies of 99.980%, 99.972%, and 98.206%, and each flagged the change point when a test sequence switched from genuine to impostor data. Quantization made the model smaller but slower in our measurements. The genuine and impostor distributions were well separated, so these results show detection of a controlled distribution shift and should not be read as evidence of general device authentication.