cs.AINov 16, 2026

Dynamic Shaping: Acquiring Interfaces and Constructing Execution over Frozen Pretrained Components

Authors: Nathaniel Oh

Abstract

We introduce Dynamic Shaping, a method for constructing executable neural arrangements through acquired response and partial-derivative interfaces over frozen pretrained components. The method learns local descriptions of available computation, ranks component rebindings using task-gradient alignment, transports the current interface across a rebinding, and retains changes through behavioral feedback. Continuous interface acquisition uses proximal policy optimization; the source coefficients remain fixed. We derive an exact identity relating transport error in component responses and partials to changes in action logits and their sensitivities. A pixels-only ViZDoom Basic study evaluates first task acquisition without a competent reference controller, demonstrations or intact response targets. Across three initializations and 100 shared held-out game seeds, initial arrangements succeed on 86/300 controller/episode pairs; joint interface-and-binding construction succeeds on 295/300. Fixed-binding interface acquisition also reaches 295/300. Each policy executes four source fragments, with 110 acquired interface values and fixed random visual and action projections. The artifact includes all candidate trials, interventions and costs; all 1,200 recorded initial/retained outcomes replay exactly. The contribution is an explicit profile, rebind, transport and retain procedure and a reproducible reference-free acquisition case over frozen pretrained computation.

Figures & tables

Explore similar work

Sep 26, 2026

Action Shaping: Policies Absorb What They Can Express

Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it action shaping and state its principle. A trainable policy absorbs an offset its own output layer can reproduce exactly, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. The condition is exact reproduction, not capacity: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. Policies absorb what they can express, and only that.
May 5, 2026

Discovering Reinforcement Learning Interfaces with Large Language Models

Reinforcement learning systems rely on environment interfaces that specify observations and reward functions, yet constructing these interfaces for new tasks often requires substantial manual effort. While recent work has automated reward design using large language models (LLMs), these approaches assume fixed observations and do not address the broader challenge of synthesizing complete task interfaces. We study RL task interface discovery from raw simulator state, where both observation mappings and reward functions must be generated. We propose LIMEN (Code available at https://github.com/Lossfunk/LIMEN), a LLM guided evolutionary framework that produces candidate interfaces as executable programs and iteratively refines them using policy training feedback. Across novel discrete gridworld tasks and continuous control domains spanning locomotion and manipulation, joint evolution of observations and rewards discovers effective interfaces given only a trajectory-level success metric, while optimizing either component alone fails on at least one domain. These results demonstrate that automatic construction of RL interfaces from raw state can substantially reduce manual engineering and that observation and reward components often benefit from co-design, as single-component optimization fails catastrophically on at least one domain in our evaluation suite.
Sep 30, 2026

Code to Control: Synthesizing Parameterized Reactive Controllers

Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.