Organizations: Department of Electrical and Computer Engineering, University of California, Riverside · Department of Computer Science, University of Colorado, Boulder · Department of Mathematics, University of California, Riverside
In this work, we present a test-driven, agentic framework for synthesizing a deployable low-level robot controller for navigation tasks. Given a 2D map with an image of an ultrasonic sensor-based robot, or a 3D robotic simulation environment, our framework iteratively refines the generated controller code using diagnostic feedback from structured test suites to achieve task success. We propose a dual-tier repair strategy to refine the generated code that alternates between prompt-level refinement and direct code editing. We evaluate the approach across 2D navigation tasks and 3D navigation in the Webots simulator. Experimental results show that test-driven synthesis substantially improves controller reliability and robustness over one-shot controller generation, especially when the initial prompt is underspecified. The source code and demonstration videos are available at: https://shivanshutripath.github.io/robotic_controller.github.io.
Figures & tables
Fig. 1: The figure shows map preprocessing pipeline. A user-provided map image is converted into a grayscale occupancy grid occ[y,x] , where occ[y,x]=True denotes an obstacle pixel and occ[y,x]=False denotes free space. This pipeline simultaneously generates params.json , which stores the configuration parameters to produce occupancy grid.
Fig. 2: The figure shows the overview of our test-driven agentic framework. The workflow is partitioned in to three functional domains: (1) The red block shows the inputs to code_gen.py , providing preprocessed occupancy maps with params.json and the robot image; (2) The blue block highlights the calibration loop, where code_gen.py generates controller.py from prompt.py . PyTest suite evaluates the generated controller, and repair.py updates the prompt template or updates the code based on the failures; (3) The green block defines the PyTest files, which are run via repair.py .
Fig. 3: The figure shows the inputs to code_gen.py in the 3 D Webots simulation. The generator (LLM 1 ) produces controller.py conditioned on (i) the current prompt specification prompt.py (fixed base prompt plus accumulated updates), (ii) the Webots world file empty.wbt describing the arena, robot, goal, and static obstacles, and (iii) a human task prompt specifying start and goal poses. The optimizer (LLM 2 ) uses Pytest feedback to update prompt.py across iterations.
Fig. 4: The figure shows the performance of our approach in 2 D navigation described in Section IV-A . The left figure shows the success rate across learner models. The right figure shows cumulative success vs iteration k .
Fig. 5: The figure shows the controllers performance for a 3 D navigation task in a Webots environment described in Section IV-B . The left figure shows the success rate across learner models. The right figure shows cumulative success vs iteration k .
Category
Test Target
Verification Objective
Static Contract
Compliance
Ensures required constants ( SENSOR_RANGE , etc.), imports, and main entry points are present.
Verifies A∗ path connectivity, and ensures all planned waypoints lie strictly within free-space coordinates.
Control
Check accuracy of differential-drive wheel speed commands for forward motion, turning, and waypoint tracking.
End-to-End
Schema
Validates simulation by checking return schemas for positions, speeds, and collision counts.
TABLE II: PyTests for Webots simulation
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Fig. 6: The figure shows the prompt-repair iterations in the 2 D simulation for a single run using GPT-5.2. Each iteration updates the prompt with new auto-repair rules derived from the previous failure, leading to a deployment-ready controller.
Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.
Yuyao Liu, Jiayuan Mao, David Hsu +2
Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania +1
Large Language Models (LLMs) have demonstrated impressive performance across various domains, including code generation and problem solving. However, their application in robotic control, particularly in low-level tasks that require precise manipulation, real-time feedback, and environment-dependent execution, remains limited. To address this challenge, we propose the Closed-Loop Modular Code Synthesizer framework. This framework leverages a pre-trained LLM without any task-specific fine-tuning to perform modular code planning and generation, and iteratively executes the generated code while inserting debugging probes to observe its behavior. This closed-loop structure facilitates systematic debugging and refinement, ultimately producing executable control programs. We apply the proposed framework to the calibration of an RGB-D camera and a robotic arm, validating its effectiveness in real-world settings. Furthermore, through a subsequent pick-and-place task, we demonstrate not only the accuracy of the calibration but also the potential extensibility of the framework. Across both tasks, the framework achieved high execution accuracy and autonomy, illustrating the practicality and scalability of LLM-based robotic control using our framework.
Gina Yoon, Sumin Lee, Joo Yong Sim
Department of Mechanical Systems Engineering, Sookmyung Women’s University, Cheongpa-ro 47-gil 100, Yongsan-gu, Seoul, 04310, South Korea
We present DreamToNav, a novel autonomous robot framework that uses generative video models to enable intuitive, human-in-the-loop control. Instead of relying on rigid waypoint navigation, users provide natural language prompts (e.g. ``Follow the person carefully''), which the system translates into executable motion. Our pipeline first employs Qwen 2.5-VL-7B-Instruct to refine vague user instructions into precise visual descriptions. These descriptions condition NVIDIA Cosmos 2.5, a state-of-the-art video foundation model, to synthesize a physically consistent video sequence of the robot performing the task. From this synthetic video, we extract a valid kinematic path using visual pose estimation, robot detection and trajectory recovery. By treating video generation as a planning engine, DreamToNav allows robots to visually "dream" complex behaviors before executing them, providing a unified framework for obstacle avoidance and goal-directed navigation without task-specific engineering. We evaluate the approach on both a wheeled mobile robot and a quadruped robot in indoor navigation tasks. DreamToNav achieves a success rate of 76.7%, with final goal errors typically within 0.05-0.10 m and trajectory tracking errors below 0.15 m. These results demonstrate that trajectories extracted from generative video predictions can be reliably executed on physical robots across different locomotion platforms.
Valerii Serpiva, Jeffrin Sam, Chidera Simon +4
The authors are with the Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology Moscow, Bolshoy Boulevard 30, bld. 1, 121205, Moscow, Russia.