EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware
Organizations: Department of Electrical and Computer Engineering, Duke University · Tesla STEM High School
Abstract
Modern coding agents perform well on general software benchmarks because they can run code and unit tests in a closed loop. Embedded firmware development, however, breaks this loop: a program that compiles can still fail on real hardware due to pin polarity, bus protocol, timing, or networking bugs that an agent at a text terminal cannot observe. Existing embedded coding agents either skip hardware execution, drive GPIO-only simulators with multi-second delays, or test on physical boards in a static environment. In this paper, we present EmbedGym, an interactive virtual hardware environment that lets coding agents develop and debug embedded firmware without physical hardware. EmbedGym uses instruction-level microcontroller unit (MCU) emulators to execute the same binary that would be flashed onto a physical board, connects them to 41 peripheral models, an analog circuit simulator, and virtual WiFi/Bluetooth Low Energy (BLE) endpoints, and exposes an interface for agents to change sensor inputs, press buttons, and inspect hardware states. On top of it, we build EmbedGym-Bench, which consists of 598 tasks across 6 MCU-framework pairs graded by hidden timed test vectors, using a five-stage multi-agent workflow that synthesizes peripheral models and tasks from datasheets and driver libraries. We expose EmbedGym to a coding agent at three levels and evaluate eight models over more than 14K trajectories. Even when compilation succeeds 99.7% of the time, 19.9% of compiled firmwares fail at runtime (up to 73% on individual targets). In contrast, interactive hardware testing in EmbedGym increases pass rates across models by +4.9% to +14.7% (up to +31% on individual targets). Our results also show that an agent with EmbedGym outperforms prior embedded agent frameworks, and lightweight models with EmbedGym match frontier models without it at 30-70x lower cost.
Figures & tables
| Hardware & Simulation Scope | Evaluation & Scaling | |||||||
|---|---|---|---|---|---|---|---|---|
| Platform / Benchmark | MCU/Boards | Frameworks | Peripherals | Execution | Timing Res. | Evaluation Oracle | Agent Feedback | Scaling |
| IoTPilot ( Gong et al., 2025 ) | – | RIOT, Zephyr, Contiki, FreeRTOS | 3 (I2C, GPIO, WiFi/UDP) | Compile only | – | Compile + static API | Compile only | Manual |
| AutoEmbed ( Yang et al., 2026 ) | ATmega328P, Nucleo-L4, RP2040, ESP32 | Arduino | 71 (I2C, SPI, UART, 1-Wire, Analog, GPIO) | Physical only | – | Static API + self-print log | Passive serial logging | Manual |
| EmbedAgent ( Xu et al., 2026 ) | ATmega328P, ESP32, RPi Pico | Arduino, ESP-IDF, MicroPython | 9 (GPIO only) | Closed-source sim (Wokwi) | 2,000 ms | GPIO pin state | 1-Shot / Build-Fix | Manual |
| IoT-SkillsBench ( Li et al., 2026 ) | ATmega2560, ESP32-S3, nRF52840 | Arduino, ESP-IDF, Zephyr | 23 (I2C, SPI, UART, 1-Wire, Analog, GPIO) | Physical only | – | Physical HIL check | 1-Shot (no feedback) | Manual |
| EmbedGym (this work) | ATmega328P, ATmega2560, ESP32-S3, nRF52840 | Arduino, ESP-IDF, Zephyr | 41 (I2C, SPI, UART, 1-Wire, Analog, GPIO, WiFi, BLE) | Deterministic sim + SPICE + physical | 0.1 ms | Timed hardware assertions | Active stimulus | Self- evolving |
| Component | Language(s) | LoC |
| avr8js (AVR) wrapper & bridge | TS | 1.2K |
| qemu (ESP32-S3) patch & bridge | C, Python | 2.1K |
| renode (nRF52840) patch & bridge | C#, Python, TS | 2.2K |
| Shared MCU worker bridge | TS, Python | 1.4K |
| 41 peripheral models | Python, TS | 11.6K |
| Peripheral datasheet verification suite | Python | 6.8K |
| Physical Target | Emulator | Sampled | Sim Pass / Real Pass | Sim Fail / Real Fail | Agree- ment |
|---|---|---|---|---|---|
| ATmega328P | avr8js | 10 | 5/5 | 5/5 | 100% |
| ATmega2560 | avr8js | 10 | 5/5 | 5/5 | 100% |
| ESP32-S3 | qemu | 20 | 10/10 | 10/10 | 100% |
| nRF52840 | renode | 20 | 10/10 | 9/10 | 95.0% |
| Total | 60 | 30/30 | 29/30 | 98.3% |
Software Engineering Agents
Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.
See all 378 papers