cs.AISep 30, 2026

Code to Control: Synthesizing Parameterized Reactive Controllers

Authors: Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates, Samuel J. Gershman

Organizations: Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.

Figures & tables

Figure 1: Code to Control schema. The LLM synthesizes the programmatic structure of the controller, while numerical parameters can be fit separately from environment feedback when useful, especially in continuous-action domains. The synthesized structure may be a complete executable controller or a parameterized representation whose numerical parameters are optimized separately. In our continuous-control experiments, we instantiate this representation as a feature map ϕ(o,t)\phi(o,t) with a fitted matrix WW mapping features to actuator commands.
Figure 2: Atari evaluation. A : Median best score over three seeds. B : Total LLM tokens. C : Pong performance and interaction cost up to the reported controller; PoE-World additionally uses a 650-step human demonstration. D : Median decision latency on a log scale, with 95% confidence intervals. † WorldCoder’s planning time is divided by the actions of the episode in which it planned. Its error bar is the range of its three runs.
Figure 3: Flappy Bird evaluation. A : Base-game performance; ∗ score is cap-limited. B : Survival under eight dynamics changes, zero-shot (open) and after parameter refitting (filled).
DomainCode to ControlGIF-MCTSWorldCoderCQLOracle
Hopper1,155.4056.68.1137.4229.1
HalfCheetah3,958.95−- 87.4−- 69.4−- 1.3893.3
Walker2d1,388.5431.98.5278.0334.7
Swimmer320.53−- 11.2−- 18.728.4317.8
Humanoid472.92153.9299.2393.31,860.7
Ant1,003.93448.4396.6998.01,304.7
Figure 4: MuJoCo continuous-control evaluation. Episode return, reported as the median over independent runs. GIF-MCTS and WorldCoder use their published budget of 30 LLM calls per environment, compared with 6 for Code to Control. Further evaluation details are described in Section 3.4 .

Explore similar work

CardsList

  1. RHO: Your Coding Agent is Secretly a Roboticist

    Jun 15, 2026Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade +3Coding AgentsProduction Agentic Systems

    University of California, Berkeley · Embodied Science · AMD

  2. PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control

    Aug 25, 2026Suhwan Choi, Jaeyoon Jung, Sungkyung Kim +2Multimodal Large Language ModelsCognitive Science

    Seoul National University · Georgia Institute of Technology

  3. GPC: Large-Scale Generative Pretraining for Transferable Motor Control

    Jun 28, 2026Yi Shi, Yifeng Jiang, Chen Tessler +1Human Motion GenerationFull-Body Humanoid Control

    Simon Fraser University, Canada and NVIDIA, USA · NVIDIA, USA · Simon Fraser University, Canada and NVIDIA, Canada