cs.SENov 16, 2026

EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware

Authors: Minghui Zhao, Yiming Li, Sophie Zhu, Maxwell Khor, Lauren Ouyang, Hai "Helen" Li, Yiran Chen, Tingjun Chen

Organizations: Department of Electrical and Computer Engineering, Duke University · Tesla STEM High School

Abstract

Modern coding agents perform well on general software benchmarks because they can run code and unit tests in a closed loop. Embedded firmware development, however, breaks this loop: a program that compiles can still fail on real hardware due to pin polarity, bus protocol, timing, or networking bugs that an agent at a text terminal cannot observe. Existing embedded coding agents either skip hardware execution, drive GPIO-only simulators with multi-second delays, or test on physical boards in a static environment. In this paper, we present EmbedGym, an interactive virtual hardware environment that lets coding agents develop and debug embedded firmware without physical hardware. EmbedGym uses instruction-level microcontroller unit (MCU) emulators to execute the same binary that would be flashed onto a physical board, connects them to 41 peripheral models, an analog circuit simulator, and virtual WiFi/Bluetooth Low Energy (BLE) endpoints, and exposes an interface for agents to change sensor inputs, press buttons, and inspect hardware states. On top of it, we build EmbedGym-Bench, which consists of 598 tasks across 6 MCU-framework pairs graded by hidden timed test vectors, using a five-stage multi-agent workflow that synthesizes peripheral models and tasks from datasheets and driver libraries. We expose EmbedGym to a coding agent at three levels and evaluate eight models over more than 14K trajectories. Even when compilation succeeds 99.7% of the time, 19.9% of compiled firmwares fail at runtime (up to 73% on individual targets). In contrast, interactive hardware testing in EmbedGym increases pass rates across models by +4.9% to +14.7% (up to +31% on individual targets). Our results also show that an agent with EmbedGym outperforms prior embedded agent frameworks, and lightweight models with EmbedGym match frontier models without it at 30-70x lower cost.

Figures & tables

Explore similar work

CardsList
  1. Closed-loop evaluation of LLM agents for embedded software development

    Oct 8, 2026Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté +1AI Coding AgentsLLM Agent Evaluation

  2. LoopHIL: Looping Hardware into LLM-assisted Agents for Wireless Embedded Systems

    Oct 16, 2026Dhairya Shah, Spanddhana Sara, Savitha Viswanadh Kandala +1LLM Agent EvaluationLLM Agent Harnesses

  3. Skilled AI Agents for Embedded and IoT Systems Development

    Mar 20, 2026Yiming Li, Yuhan Cheng, Mingchen Ma +6AI Coding AgentsAI Agent Benchmarks