LoopHIL: Looping Hardware into LLM-assisted Agents for Wireless Embedded Systems

Dhairya Shah; Spanddhana Sara; Savitha Viswanadh Kandala; Ambuj Varshney

cs.NIOct 16, 2026

LoopHIL: Looping Hardware into LLM-assisted Agents for Wireless Embedded Systems

Authors: Dhairya Shah, Spanddhana Sara, Savitha Viswanadh Kandala, Ambuj Varshney

Organizations: National University of Singapore, Singapore

Abstract

LLM agents are widely used for software development and debugging, but their adoption in embedded systems, especially for wireless tasks, remains limited. This is because embedded development requires hardware-in-the-loop testing. Agents cannot directly observe the spectrum, probe pins, or measure power, so they cannot verify wireless link functionality; currently, human engineers perform this verification. We systematically evaluate whether LLM agents can establish wireless links when provided with hardware access. We introduce LoopHIL, a framework that lets agents control a multi-device RF bench, monitor it with a spectrum analyzer, logic analyzer, and energy meter, and validate links against a verification ladder, all within code-enforced safety limits. Across four increasingly complex tasks, from generating a carrier to backscatter decoding by a commercial receiver, and using six models from two vendors, hardware-in-the-loop increases task completion rates from 54% to 79% and from 17% to 83% on the most challenging task. It also halves task time and output tokens. With debugging tools, agents decrease tag power consumption by up to 96%. However, small models often report unverified claims, so independent hardware checks remain necessary.

Figures & tables

Figure 1. LoopHIL overview . The agent reaches the physical layer only through LoopHIL : typed tools with safety limits in code drive the RF bench, and the instruments return features to a verification ladder that blames the faulty device.

Figure 2. LoopHIL architecture . The agent calls typed tools that drive the RF bench under safety limits in code; the instruments feed measurements to the checks, which return a verdict to the agent.

Figure 3. LoopHIL hardware test bench used for evaluation.

No.	Task Description	Level
T1	CW carrier. Transmit an unmodulated carrier at 868 MHz and 0 dBm; capture 5 s at the receiver and plot received signal strength vs. time.	Easy
T2	OOK link. Transmit OOK packets with payload 0x6F379BCDE6F379 at 10 kb/s, 915 MHz and 10 dBm transmit power; decode 100 packets, report the BER, and plot a 2 s waterfall spectrum.	Medium
T3	OOK backscatter. Implement OOK backscatter on a 100 kHz baseband subcarrier at 20 kbps on a MSP430FR5969 under a 915 MHz carrier (10 dBm transmit power); decode 100 packets with payload 0x6F379BCDE6F379 , report the BER, and plot one received spectrum.	Medium
T4	2-FSK backscatter. Generate 2-FSK on 800 and 1000 kHz subcarriers at 100 kb/s in a CC1310-compatible frame under a 915 MHz carrier (10 dBm transmit power); receive 500 packets with payload 0x6F379BCDE6F379 on the CC1310, save its output, report the BER, and plot one received spectrum.	Hard

Table 1. Benchmark tasks. All tasks use two USRP B210s (TX and RX); T3 and T4 add an MSP430FR5969 tag, and T4 a TI CC1310 receiver.

	Frontier	Mid	Small
Anthropic	Claude Opus 5.5	Claude Sonnet 5.5	Claude Haiku 4.5
OpenAI	GPT-6-Astra	GPT-5.6-Sol	GPT-5.6-Luna

Table 2. Models evaluated, by vendor and tier.

		Anthropic			OpenAI
Task	Setting	Opus 5.5	Sonnet 5.5	Haiku 4.5	GPT-6 Astra	GPT-5.6 Sol	GPT-5.6 Luna
T1	B	✓	✓	✗ †	✓	✓	✗ †
T1	H	✓	✓	✓	✓	✓	✗
T2	B	✓	✓	✗ †	✓	✓	✓
T2	H	✓	✓	✗ †	✓	✓	✓
T3	B	✓	✗	✗ †	✓	✓	✗ †

Table 3. Task completion per model. B: baseline; H: harness.

\dagger

: unsupported claim. Highlighted: completed only with the harness.

Figure 5. Wall-clock time and API cost of each completed run. Colour: vendor; shape: tier; outline: baseline; filled: harness.

Task	Completed	Speedup ∗	Cost lower by ∗	Tokens lower by ∗
T1	67% $\,\rightarrow\,$ 83%	2.0 $\times$	1.3 $\times$	2.1 $\times$
T2	83% $\,\rightarrow\,$ 83%	1.9 $\times$	1.1 $\times$	1.7 $\times$
T3	50% $\,\rightarrow\,$ 67%	2.3 $\times$	1.6 $\times$	2.1 $\times$
T4	17% $\,\rightarrow\,$ 83%	0.4 $\times$ †	0.3 $\times$ †	0.4 $\times$ †

Table 4. Effect of the harness per task (baseline

\rightarrow

harness). Factors are medians over models that complete the task in both conditions.

Figure 6. (a) Completion without and with hardware debugging tools. (b) Tag power of the first working and the final firmware in T4 + energy. B: baseline; H: harness; ✓: task completed.

Panel of Figure 7 — Figure 7. One run of T3 + debug (Opus 5.5, harness), which decodes 250 packets without bit errors. (a) One OOK packet captured from the logic analyzer. (b) Frequency spectrum showing the modulated carrier with sidebands at a 100 kHz offset.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8. Prompt used for Task T4 + energy optimization.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

Skilled AI Agents for Embedded and IoT Systems Development
Mar 20, 2026Yiming Li, Yuhan Cheng, Mingchen Ma +6AI Coding Agents AI Agent Benchmarks
Department of Electrical and Computer Engineering, Duke University
Closed-loop evaluation of LLM agents for embedded software development
Oct 8, 2026Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté +1AI Coding Agents LLM Agent Evaluation
Lucentia Research, Department of Software and Computing Systems, University of Alicante, Ctra. de San Vicente del Raspeig s/n, 03690 Sant Vicent del Raspeig, Spain · Universitary Institute of Materials Technology (IUTM), Universitat Politècnica de València, Plaça Ferrándiz i Carbonell, 03801 Alcoi, Spain
HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
Apr 16, 2026Fan Cui, Hongyuan Hou, Zizhang Luo +2Automated Program Repair Electronic Design Automation
Peking University · Beijing, China

Mar 20, 2026J/K move · Enter open · S save

Skilled AI Agents for Embedded and IoT Systems Development

Yiming Li, Yuhan Cheng, Mingchen Ma, Yihang Zou+5

Department of Electrical and Computer Engineering, Duke University

AI Coding Agents AI Agent Benchmarks Internet of Things LLM Agent Skill Learning

Open paper AskPDF arXiv

Large language models (LLMs) and agentic systems have shown promise for automated software development, but applying them to hardware-in-the-loop (HIL) embedded and Internet-of-Things (IoT) systems remains challenging due to the tight coupling between software logic and physical hardware behavior. Code that compiles successfully may still fail when deployed on real devices because of timing constraints, peripheral initialization requirements, or hardware-specific behaviors. To address this challenge, we introduce a skills-based agentic framework for HIL embedded development together with IoT-SkillsBench, a benchmark designed to systematically evaluate AI agents in real embedded programming environments. IoT-SkillsBench spans three representative embedded platforms, 23 peripherals, and 42 tasks across three difficulty levels, where each task is evaluated under three agent configurations (no-skills, LLM-generated skills, and human-expert skills) and validated through real hardware execution. Across 378 hardware validated experiments, we show that concise human-expert skills with structured expert knowledge enable near-perfect success rates across platforms.

Abstract

LLM Agent Evaluation

Explore similar work

Skilled AI Agents for Embedded and IoT Systems Development

Closed-loop evaluation of LLM agents for embedded software development

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks