Reinforcement learning has achieved strong performance in many tasks by letting agents learn from nothing more than a reward signal. However, in more complex tasks, agents still struggle to learn good, generalizable policies, even with carefully designed rewards and abundant samples. We argue that this difficulty stems from the reward decoding burden: designers encode their knowledge of the desired behavior into rewards and penalties, and agents must infer this knowledge from long sequences of noisy scalars. To address this burden, we propose prescriptive policy learning (PPL), which directly teaches agents higher-level knowledge about the tasks, termed principles of behavior, such as how to maintain balance, how to accumulate energy, and how to land smoothly. Through appropriately selected key quantities and references, PPL can directly indicate at every step whether the agent follows the principles and to what extent. Our experiments show that agents can learn good, generalizable policies without any reward. Taught only simple heuristic principles, PPL agents outperform reward-based methods with far fewer samples. Their policies also transfer directly to harder settings and keep performing well, demonstrating their strong generalization ability. In summary, PPL provides a new approach in which rough knowledge of the desired behavior is enough for agents to learn good, generalizable policies, even when the system dynamics cannot be accurately modeled.
Figures & tables
Figure 1: Illustration of the reward decoding burden (Lunar Lander with wind).
Figure 2: Illustrations of different tasks.
Task
Principle
Key quantity
Reference
Obstacle avoidance
Stay as far from the barriers as possible.
q=v , the vertical velocity
v , the vertical velocity that a conservative deceleration brings to zero at the height of the gap center
Lunar Lander
Stay upright, approach the pad, and avoid descending too fast.
q=(vy,ω,vx) , the vertical, angular, and horizontal velocities
vy , ω , and vx , computed from the height above the pad, the tilt, and the horizontal position and velocity
Reacher
Turn each joint toward the configuration that places the fingertip on the target.
q=(θ˙1,θ˙2) , the joint angular velocities
qd , computed from the angle that joint d has left to turn
Table 1: Principles formulated for three tasks, with their key quantities and references.
Figure 3: Mechanism of PPL in Lunar Lander with wind.
Success (%)
Steps to 90% ( ×103 )
Clearance
Method
Moderate
Strong
Moderate
Strong
Moderate
Strong
PPL
100.0±0.0
100.0±0.0
4.24
4.18
0.166
0.155
PPO
100.0±0.2
99.8±0.5
10.0
10.0
0.112
0.107
DQN
87.1±24.3
96.6±2.3
75.0
85.0
0.097
0.087
PPO-R
99.0±1.4
96.0±2.2
20.0
20.0
0.053
0.048
Table 2: Results in obstacle avoidance under moderate and strong wind.
Figure 4: Training success rate in obstacle avoidance.
Figure 5: A successful PPL episode in obstacle avoidance under moderate wind: (a) trajectory, barriers, and center of the next gap; (b) vertical velocity and its reference against horizontal position.
Success (%)
Steps to 90% ( ×103 )
Method
Without wind
With wind
Without wind
With wind
PPL
100.0±0.2
84.6±3.0
19.7
29.3
PPO
98.8±1.5
81.2±26.3
520
790
DQN
57.2±30.5
25.2±28.0
135
390
Discrete SAC
0.0±0.0
0.0±0.0
–
–
PPO-R
0.1±0.1
0.0±0.0
–
–
Table 3: Results in Lunar Lander without and with wind. A dash marks a median seed that does not reach 90% within its budget.
Figure 6: Training success rate in Lunar Lander.
Figure 7: A successful PPL landing with wind: (a) trajectory and orientations; (b) states over time, with velocity references (dashed), the pad center and the target tilt (dotted), and the period after first ground contact (shaded).
Method
Standard
Coupled joints
Gravity
Gravity and disturbances
PPL
99.2±0.6
99.1±0.5
99.3±0.8
95.2±2.0
PPO
68.2±14.2
75.9±9.8
14.7±7.4
3.1±1.2
SAC
98.7±1.2
98.5±1.3
37.9±5.1
9.8±3.4
PPO-R
0.9±1.5
1.4±1.7
0.6±0.8
1.9±1.4
Principle controller, best gain
100.0
100.0
69.0
59.0
Table 4: Final success rate (%) in the four Reacher settings.
Figure 8: Training success rate in the four Reacher settings.
Figure 9: A successful PPL episode in Reacher with gravity and disturbances: (a) fingertip trajectory, arm poses, and target configuration; (b) joint angular velocities and their references; (c) distance to the target and the 0.02m success threshold.
Modern AI systems such as self-driving cars and game-playing agents can achieve superhuman performance, but often lack human-like generalization, interpretability, and inter-operability with human users. Inspired by the rich interactions between language and decision-making in humans, we introduce Policy Learning with a Language Bottleneck (PLLB), a framework enabling AI agents to generate linguistic rules that capture the high-level strategies underlying rewarding behaviors. PLLB alternates between a rule generation step guided by language models, and an update step where agents learn new policies guided by rules, even when a rule is insufficient to describe an entire complex policy. Across five diverse tasks, including a two-player signaling game, maze navigation, image reconstruction, and robot grasp planning, we show that PLLB agents are not only able to learn more interpretable and generalizable behaviors, but can also share the learned rules with human users, enabling more effective human-AI coordination. We provide source code for our experiments at https://github.com/meghabyte/bottleneck .
Megha Srivastava, Cedric Colas, Dorsa Sadigh +1
Stanford University · Massachusetts Institute of Technology Inria
Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions. Recent work, such as Eureka, automates reward design by using an LLM to iteratively generate and refine reward code from task descriptions. However, they rely on coarse feedback signals such as success rate, which provide little semantic insight into the learned behavior. As a result, their trained policies achieve the final goal but are frequently poorly aligned with task instructions. We introduce the Reward Design Agent (RDA), a VLM-based agentic framework that injects semantic understanding into reward design. RDA decomposes tasks, visually evaluates trajectories, summarizes failure modes, and iteratively revises reward code to better align with task instructions. Across 12 tabletop manipulation tasks from ManiSkill and 4 whole-body manipulation tasks from HumanoidBench, RDA produces policies substantially more instruction-aligned than those of other baselines, while achieving comparable task success rates. Videos and the generated reward code are available on https://nitinkamra1992.github.io/reward-design-agent.
Robots that learn from a few demonstrations often require two forms of generalization. Compositional generalization recombines skills to solve new tasks, and skill generalization lets the learned policy behind each skill work in new situations. The two depend on each other, yet information is lost between composition and the skills it calls. Where a skill works is determined by the structure its policy is trained with, while composition sees the skill only through a separate description, such as a name, an instruction, or a symbolic operator, that omits this structure. Our key idea is to use each policy's structural prior as part of the interface between composition and the skill. A structural prior states what a behavior depends on, for example that a grasp depends only on the gripper's pose relative to the object. Built into training, it shapes where the policy generalizes; stated in language, it tells composition where the policy applies. We instantiate this idea in Agent Priors-guided Policy Learning (APPL). A construction agent segments complete demonstrations into reusable skills, proposes several structural priors for each skill, and trains and verifies one policy per prior. A runtime agent then selects among these prior-specific policies and composes them toward new task goals using their interfaces. Across MetaWorld and long-horizon ManiSkill tasks, APPL improves out-of-distribution skill generalization and enables previously unseen skill compositions; ablating the interface information substantially reduces performance. These results support the use of training-time structural assumptions as a bridge between skill learning and skill composition.