Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications

Zhiqin Qian; Ryan Diaz; Sangwon Seo; Vaibhav Unhelkar

cs.AIFeb 20, 2026

Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications

Authors: Zhiqin Qian, Ryan Diaz, Sangwon Seo, Vaibhav Unhelkar

Organizations: Rice University, Houston, TX, USA

Abstract

When training artificial intelligence (AI) to perform tasks, humans often care not only about whether a task is completed but also how it is performed. As AI agents tackle increasingly complex tasks, aligning their behavior with human-provided specifications becomes critical for responsible AI deployment. Reward design provides a direct channel for such alignment by translating human expectations into reward functions that guide reinforcement learning (RL). However, existing methods are often too limited to capture nuanced human preferences that arise in long-horizon tasks. Hence, we introduce Hierarchical Reward Design from Language (HRDL): a problem formulation that extends classical reward design to encode richer behavioral specifications for hierarchical RL agents. We further propose Language to Hierarchical Rewards (L2HR) as a solution to HRDL. Experiments show that AI agents trained with rewards designed via L2HR not only complete tasks effectively but also better adhere to human specifications. Together, HRDL and L2HR advance the research on human-aligned AI agents.

Figures & tables

Figure 1. This work introduces the Hierarchical Reward Design from Language (HRDL) problem. Unlike prior work on reward design, HRDL decomposes reward design into low- and high-level components $(\tilde{r}_{L},\tilde{r}_{H})$ . Language to Hierarchical Rewards (L2HR), our proposed solution to HRDL, leverages language models to guide the synthesis of these hierarchical rewards, enabling existing RL algorithms to train agents that are better aligned with human specifications.

Panel of Figure 2 — Figure 2. Although prior works have utilized multi-level rewards to train hierarchical agents, the design of such rewards remains underexplored and lacks a concrete problem formulation. This work bridges this gap by introducing HRDL.

Panel of Figure 3 — Figure 3. Illustration of L2HR input and output.

Panel of Figure 4 — Figure 4. Rendered scenes from the iTHOR and Kitchen domains. Fig. 1 (right) illustrates the Rescue World domain.

Domain	Method	High-Level		Low-Level		Total
Domain	Method	Rewards $\uparrow$	Expert % $\uparrow$	Rewards $\uparrow$	Expert % $\uparrow$	Rewards $\uparrow$	Expert % $\uparrow$
Rescue	Task	11.22 $\pm$ 5.57	20.00	-16.46 $\pm$ 5.49	0.00	73.80 $\pm$ 5.70	0.00
	Flat	9.38 $\pm$ 7.02	12.50	-2.62 $\pm$ 5.19	62.50	85.13 $\pm$ 9.33	12.50
	Hier	16.65 $\pm$ 6.93	76.92	-0.69 $\pm$ 1.58	76.92	93.98 $\pm$ 9.01	69.23
iTHOR	Task	4.10 $\pm$ 1.34	0.00	-23.38 $\pm$ 1.86	0.00	12.31 $\pm$ 0.66	0.00
	Flat	7.67 $\pm$ 4.48	0.00	-35.20 $\pm$ 12.42	0.00	3.27 $\pm$ 9.31	0.00

Table 1. Table showing the performance of policies trained with the task reward alone or combined with LLM-generated flat or hierarchical rewards. For each metric, we report both the cumulative reward returns and the percentage of policies at expert-level alignment (attaining the maximum possible cumulative return for that metric). Means and standard deviations are computed over all runs for the Task baseline, and only over the LLM-generated reward candidates that successfully complete the task for Flat and Hier .

Figure 5. Syntax error rates for LLM-based reward generation, computed over 24 candidates per configuration.

Figure 6. Task completion rates for LLM-generated rewards, calculated as the proportion of designed rewards that preserve task feasibility among syntactically valid candidates.

Figure 7. Human-provided ratings for agent alignment.

	Rescue World			Kitchen
Method	Persistence $\uparrow$	Safety $\uparrow$	Overall $\uparrow$	Chopping $\uparrow$
Flat	12.50%	50.00%	12.50%	10.00%
Hier	76.92%	61.54%	53.85%	71.43%

Table 2. Candidate agent policies (%) that received a perfect alignment score from all human participants.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8. Screenshots of renderings of the three task domains used in our study.

Domain	Method	High-Level		Low-Level		Total
Domain	Method	Rewards $\uparrow$	Expert % $\uparrow$	Rewards $\uparrow$	Expert % $\uparrow$	Rewards $\uparrow$	Expert % $\uparrow$
Rescue	Task	11.22 $\pm$ 5.57	20.00	-16.46 $\pm$ 5.49	0.00	73.80 $\pm$ 5.70	0.00
	Flat*	6.84 $\pm$ 5.24	0.00	-0.32 $\pm$ 0.72	80.00	85.88 $\pm$ 4.66	0.00
	Hier*	20.00 $\pm$ 0.00	100.00	-1.00 $\pm$ 2.24	80.00	97.15 $\pm$ 2.94	80.00
iTHOR	Task	4.10 $\pm$ 1.34	0.00	-23.38 $\pm$ 1.86	0.00	12.31 $\pm$ 0.66	0.00
	Flat*	7.80 $\pm$ 2.41	0.00	0.00 $\pm$ 0.00	100.00*	35.41 $\pm$ 2.98	0.00

Appendix

Table 3. Table showing the performance of policies trained with the task reward alone and with task reward combined with expert-provided flat or hierarchical rewards. For each metric (“High-Level,” “Low-Level,” and “Total”), we report both the cumulative returns and the percentage of policies achieving expert-level alignment. A policy is considered expert-aligned at the high or low level if it attains the maximum possible cumulative return for that metric. “Total” represents the sum of the task reward, high-level alignment reward, and low-level alignment reward, and a policy is deemed expert-level overall if it aligns at both levels.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 261 papers

RDA: Reward Design Agent for Reinforcement Learning
Jun 1, 2026Hojoon Lee, Ajay Subramanian, Ben Abbatematteo +4Agentic RL Reinforcement Learning
Meta · Holiday Robotics · Boston Dynamics
Specifying Reward Functions for RL Without Environment Sampling
Sep 14, 2026Stephane Hatgis-Kessell, W. Bradley Knox, Emma BrunskillLLM-Guided RL Reward Design for RL
1Computer Science Department, Stanford University · 2Computer Science Department, The University of Texas at Austin
Inverse RL Helps Align AI by Imitating Humans
Jul 27, 2026Michał Wiliński, Liu Leqi, Chirag NagpalRL for Language Models LLM Alignment
Carnegie Mellon University · The University of Texas at Austin · Independent Researcher

Jun 1, 2026J/K move · Enter open · S save

RDA: Reward Design Agent for Reinforcement Learning

Hojoon Lee, Ajay Subramanian, Ben Abbatematteo, Vijay Veerabadran+3

Meta · Holiday Robotics · Boston Dynamics

Agentic RL Reinforcement Learning Robot Skill Learning Robotic RL Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions. Recent work, such as Eureka, automates reward design by using an LLM to iteratively generate and refine reward code from task descriptions. However, they rely on coarse feedback signals such as success rate, which provide little semantic insight into the learned behavior. As a result, their trained policies achieve the final goal but are frequently poorly aligned with task instructions. We introduce the Reward Design Agent (RDA), a VLM-based agentic framework that injects semantic understanding into reward design. RDA decomposes tasks, visually evaluates trajectories, summarizes failure modes, and iteratively revises reward code to better align with task instructions. Across 12 tabletop manipulation tasks from ManiSkill and 4 whole-body manipulation tasks from HumanoidBench, RDA produces policies substantially more instruction-aligned than those of other baselines, while achieving comparable task success rates. Videos and the generated reward code are available on https://nitinkamra1992.github.io/reward-design-agent.

Abstract

Reward Design for RL

Explore similar work

RDA: Reward Design Agent for Reinforcement Learning

Specifying Reward Functions for RL Without Environment Sampling

Inverse RL Helps Align AI by Imitating Humans