ROOT: Discovering Rewards for User-Specified Embodied Behaviors

Eren Sadikoglu; Aditya Taparia; Xinyuan Liu; Ransalu Senanayake

cs.ROOct 3, 2026

ROOT: Discovering Rewards for User-Specified Embodied Behaviors

Authors: Eren Sadikoglu, Aditya Taparia, Xinyuan Liu, Ransalu Senanayake

Organizations: Robotics and Autonomous Systems, Arizona State University, Tempe, Arizona, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA

Abstract

Reinforcement learning for embodied control remains constrained by the difficulty of reward specification. Although recent large language model (LLM)-based methods can synthesize reward functions from natural-language descriptions, they often fail to capture subtle behavioral properties that humans care about, such as natural gait, posture, and movement style. This limitation arises because many desired behaviors are easier to recognize visually than to encode in a reward function. We introduce Reward Optimization via Observable Trees (ROOT), a framework for discovering reward functions that align learned policies with user-specified embodied behaviors. Rather than relying solely on scalar training statistics, ROOT casts reward design as an observation-guided search over a persistent experiment tree that stores reward programs, trained policies, and rollout observations, together with behavioral insights distilled by a video-language model, to diagnose behavioral failures and guide subsequent reward refinements. We evaluate ROOT on seven tasks across four embodiments: simulated Hopper, HalfCheetah, Ant, Unitree Go2, and as well as the real-world Unitree Go2. ROOT produces behaviors that better align with user intent than those generated by existing LLM-based reward-generation methods, achieving up to 86.8% locomotion-completeness accuracy and improving Vid-LLM behavioral alignment from 3.56/5 to 4.14/5, a 16.5% improvement over baselines. Human evaluations further support these results, with ROOT preferred in 51-63% of pairwise comparisons.

Figures & tables

Fig. 1: Overview of ROOT. Given a user-specified behavior, ROOT explores a tree of candidate reward programs. Each candidate is visually audited and saved. The learned tree guides later reward generation toward policies that better match the intended behavior.

Fig. 2: Observation-guided reward search with ROOT. Each tree expansion selects a reference reward and a refinement hypothesis, generates a reward program, trains a policy, and records the observed outcome. The experience tree preserves both partial successes and unsuccessful refinements, allowing evidence from earlier experiments and other branches to condition subsequent proposals. A Vid-LLM supplies visual feedback during search.

Robot	Task	Metric	Eureka	Text2	Language	CARD	LAPP	ROOT
				Reward	2Reward			(Ours)
		Align. Score	4.67	4.50	4.61	4.67	3.94	4.67
	Walk forward with a trot gait.	Acc. (%)	$\underline{79.6\!\pm\!0.7}$	$78.1\!\pm\!0.1$	$9.8\!\pm\!0.3$	$79.5\!\pm\!0.4$	$0.0\!\pm\!0.0$	$\mathbf{86.8\!\pm\!0.3}$
		Align. Score	4.56	4.39	4.39	4.50	4.17	4.50
	Walk forward with a pace gait.	Acc. (%)	$74.9\!\pm\!0.2$	$\underline{77.5\!\pm\!0.3}$	$13.7\!\pm\!1.5$	$76.1\!\pm\!0.5$	$7.48\!\pm\!0.9$	$\mathbf{78.4\!\pm\!0.1}$
		Align. Score	2.78	2.72	2.22	3.50	3.22	3.61

TABLE I: Behavioral Alignment across Simulated Embodiments. We report task performance and Vid-LLM alignment; higher is better. Best and second-best scores are bold and underlined. Locomotion completeness accuracy uses task-specific metrics defined in Sec. IV-A and reports mean and standard deviation across three evaluation seeds. Alignment reports mean Vid-LLM scores. ∗ native score denotes mean velocity multiplied by mean survival. ROOT achieves the best behavioral alignment while maintaining comparable or higher task performance across tasks.

Fig. 3: Human preferences across pairwise comparisons of ROOT against pooled baselines on three behaviors.

Fig. 4: Qualitative rollouts of policies learned with ROOT . From top to bottom: Hopper hopping, Ant spider-creeping, and Unitree Go2 cantering in simulation, followed by a simulation-trained trotting policy deployed on a physical Unitree Go2. Frames progress from left to right.

Configuration	Metric	Ant		Unitree Go2
Configuration	Metric	Crab	Spider	Canter
ROOT (Full)	Acc. (%)	$\mathbf{47.67\!\pm\!2.64}$	$\mathbf{35.50\!\pm\!4.8}$	$\mathbf{70.5\!\pm\!4.0}$
ROOT (Full)	Align. Score	4.44	3.89	3.61
No History (No Tree)	Acc. (%)	$2.97\pm 2.4$	$9.63\pm 11.1$	$19.63\pm 0.35$
No History (No Tree)	Align. Score	2.83	2.67	2.89
No Vid-LLM and History	Acc. (%)	$0.97\pm 0.06$	$3.40\pm 2.97$	$1.37\pm 0.06$

TABLE II: We compare ROOT with variants that remove experimental history (or tree) and both history and Vid-LLM feedback during reward search. We report locomotion completeness accuracy (mean

\pm

SD across three evaluation seeds) and mean Vid-LLM alignment.) and mean Vid-LLM alignment; higher is better. ROOT results are bold.

Panel of Figure 7 — Fig. 5: Avg. behavioral alignment versus training compute (GPU-hours, left) and LLM/VLM requests (right) on Ant crab-crawling and Go2 canter. Higher alignment and lower cost are better.

Fig. 6: Foot contact timelines and foot-force signals for two Unitree Go2 locomotion gaits. (Left) Trotting on a real Unitree Go2 quadruped, showing characteristic alternating diagonal leg pairs (FL/RR and FR/RL) with regular, synchronised contact phases. (Right) Pacing in simulation, where ipsilateral leg pairs (FL/RL and FR/RR) move together, producing a distinctly different contact pattern.

Fig. 7: Gait quality metrics comparing three locomotion policies on a real Unitree Go2 robot across 7–8 trials per policy (mean $\pm$ SD). Metrics are grouped into three categories. Motion Quality (a–c): Contact Smoothness measures the regularity of foot touchdown transitions ( $\uparrow$ better); Joint Jerk is the RMS rate of change of joint acceleration in rad/s 3 ( $\downarrow$ better); Joint-motion magnitude is the RMS angular velocity across all joints in rad/s ( $\downarrow$ better). Gait Regularity (d–e): Stride Frequency is the mean stepping cadence in Hz; Cadence Entropy measures irregularity in inter-stride intervals in bits ( $\downarrow$ better). Inter-leg Coordination (f–g): Phase Consistency quantifies the stability of diagonal leg synchronization on a 0–1 scale ( $\uparrow$ better); Phase Offset is the mean phase difference between diagonal leg pairs in degrees, where a trot ideally exhibits a phase offset of approximately $0^{\circ}$ . These measurements characterize the recorded policies; joint angular velocity is a motion proxy and does not establish energy efficiency.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Task	Behavior specification
Hopper (MJX)	Forward hopping	Hop forward as fast as possible.
HalfCheetah (MJX)	Forward running	Run forward as fast as possible.
Ant (MJX)	Spider-creep	Make the ant creep forward like a spider, using all four legs in a coordinated crawling gait while keeping its body close to the surface.
Ant (MJX)	Crab-crawl	Make the ant crawl sideways like a crab, moving laterally using coordinated leg movements while keeping its body low and stable.
Go2 (Isaac Lab)	Pace	Train the Go2 to pace forward—legs on the same side (FL+RL, then FR+RR) move together in a lateral two-beat gait with steady forward velocity.
Go2 (Isaac Lab)	Trot	Train the Go2 to perform a trot gait—a two-beat diagonal pattern where diagonally opposite legs (FL+RR, then FR+RL) move together in alternation with steady forward velocity.

Appendix

TABLE III: Natural-language behavior specifications and simulator backends. The Go2 trot policy is also evaluated on a physical robot.

Parameter	Ant / Hopper (MJX)	HalfCheetah (MJX)	Go2 canter (MJX)	Go2 pace/trot (Isaac Lab)
Parallel environments	$4096$	$4096$	$8192$	$4096$
Episode limit (control steps)	$1000$	$1000$	$1000$	$1000$
Transitions per candidate	$2\times 10^{7}$	$10^{7}$	$3\times 10^{7}$	$98{,}304{,}000$
Training iterations (RSL-RL)	—	—	—	$1000$
Learning rate	$3\times 10^{-4}$	$3\times 10^{-4}$	$3\times 10^{-4}$	$10^{-3}$
Learning-rate schedule	Constant	Constant	Constant	Adaptive

Appendix

TABLE IV: PPO settings for ROOT . Ant includes both crab-crawl and spider-creep. MJX tasks use Brax PPO; Isaac Lab tasks use RSL-RL PPO. Training budgets are environment transitions per reward candidate. Dashes indicate unspecified or inapplicable settings.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 250 papers

Jun 1, 2026

RDA: Reward Design Agent for Reinforcement Learning

Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions. Recent work, such as Eureka, automates reward design by using an LLM to iteratively generate and refine reward code from task descriptions. However, they rely on coarse feedback signals such as success rate, which provide little semantic insight into the learned behavior. As a result, their trained policies achieve the final goal but are frequently poorly aligned with task instructions. We introduce the Reward Design Agent (RDA), a VLM-based agentic framework that injects semantic understanding into reward design. RDA decomposes tasks, visually evaluates trajectories, summarizes failure modes, and iteratively revises reward code to better align with task instructions. Across 12 tabletop manipulation tasks from ManiSkill and 4 whole-body manipulation tasks from HumanoidBench, RDA produces policies substantially more instruction-aligned than those of other baselines, while achieving comparable task success rates. Videos and the generated reward code are available on https://nitinkamra1992.github.io/reward-design-agent.

Hojoon Lee, Ajay Subramanian, Ben Abbatematteo +4

Meta · Holiday Robotics · Boston Dynamics

Agentic RL Reinforcement Learning Robot Skill Learning Robotic RL Reward Design for RL

Ask PDF

Sep 26, 2026

Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization

Designing dense reward functions for low-level reinforcement learning (RL) control remains difficult. Recent work uses large language models (LLMs) to iteratively generate and refine reward functions using policy-training feedback within scripted search algorithms. However, evaluating each candidate requires a full RL training run, making sample efficiency a central challenge for reward search on complex control tasks. To address this limitation, we propose an Agentic Reward Black-box Optimization (ARBO) framework, in which an LLM agent builds the search strategy at run time from an evaluation history maintained as its persistent workspace. The evaluation history comprises two components: observations maintained by the evaluation oracle, including candidate scores, per-term training curves, and error tracebacks; and an agent-maintained belief that records diagnoses and intended next steps. The agent queries both with tools and generates the next batch of reward candidates, rather than generating them in a single pass from a fixed prompt. Across four control domains, ARBO achieves gains of 29.9% in manipulation success rate and 192.8% in power-grid score over baseline means under a shared evaluation budget. Ablations examine each component's contribution and sensitivity to backbone choice.

Minghao Li, Rui Tan, Ruihang Wang

Nanyang Technological University · Yunnan University

Reward Design for RL RL Control Black-Box Optimization Sample-Efficient RL

Ask PDF

Mar 10, 2026

Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning

We introduce Reward-Zero, a general-purpose implicit reward mechanism that transforms natural-language task descriptions into dense, semantically grounded progress signals for reinforcement learning (RL). Reward-Zero serves as a simple yet sophisticated universal reward function that leverages language embeddings for efficient RL training. By comparing the embedding of a task specification with embeddings derived from an agent's interaction experience, Reward-Zero produces a continuous, semantically aligned sense-of-completion signal. This reward supplements sparse or delayed environmental feedback without requiring task-specific engineering. When integrated into standard RL frameworks, it accelerates exploration, stabilizes training, and enhances generalization across diverse tasks. Empirically, agents trained with Reward-Zero converge faster and achieve higher final success rates than conventional methods such as PPO with common reward-shaping baselines, successfully solving tasks that hand-designed rewards could not in some complex tasks. In addition, we develop a mini benchmark for the evaluation of completion sense during task execution via language embeddings. These results highlight the promise of language-driven implicit reward functions as a practical path toward more sample-efficient, generalizable, and scalable RL for embodied agents. Code will be released after peer review.

Heng Zhang, Haddy Alchaer, Arash Ajoudani +1

Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy · Ph.D. program of national interest in Robotics and Intelligent Machines (DRIM) and Universit ‘a di Genova, Genoa, Italy. · Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA. +2

Reward Design for RL Dense Reward RL Reward Shaping Sparse-Reward RL

Ask PDF

Abstract

Reward Design for RL

Explore similar work

RDA: Reward Design Agent for Reinforcement Learning

Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization

Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning