Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning

Heng Zhang; Haddy Alchaer; Arash Ajoudani; Yu She

cs.LGMar 10, 2026

Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning

Authors: Heng Zhang, Haddy Alchaer, Arash Ajoudani, Yu She

Organizations: Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy · Ph.D. program of national interest in Robotics and Intelligent Machines (DRIM) and Universit ‘a di Genova, Genoa, Italy. · Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA. · Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA. ·

Abstract

We introduce Reward-Zero, a general-purpose implicit reward mechanism that transforms natural-language task descriptions into dense, semantically grounded progress signals for reinforcement learning (RL). Reward-Zero serves as a simple yet sophisticated universal reward function that leverages language embeddings for efficient RL training. By comparing the embedding of a task specification with embeddings derived from an agent's interaction experience, Reward-Zero produces a continuous, semantically aligned sense-of-completion signal. This reward supplements sparse or delayed environmental feedback without requiring task-specific engineering. When integrated into standard RL frameworks, it accelerates exploration, stabilizes training, and enhances generalization across diverse tasks. Empirically, agents trained with Reward-Zero converge faster and achieve higher final success rates than conventional methods such as PPO with common reward-shaping baselines, successfully solving tasks that hand-designed rewards could not in some complex tasks. In addition, we develop a mini benchmark for the evaluation of completion sense during task execution via language embeddings. These results highlight the promise of language-driven implicit reward functions as a practical path toward more sample-efficient, generalizable, and scalable RL for embodied agents. Code will be released after peer review.

Figures & tables

Figure 1: Conceptual comparison of human learning, traditional RL, and the proposed Reward-Zero. The left panel illustrates how Human Learning is intuitive and implicit, driven by visual matching and a generalized "Sense of Completion," without relying on explicit rewards, mathematical environment models, or exact object world coordinates. The center panel depicts Traditional RL training a simple robotic policy, a rigid and explicit approach that typically requires a hand-crafted reward function, a precise environment model, heavy observation and exact object coordinates to function. The right panel introduces Reward-Zero, our flexible and language-driven method that represents a sophisticated universal reward function. This mechanism uses a general-purpose language embedding-driven implicit reward mechanism to generate a continuous sense-of-completion signal by comparing task and experience embeddings. As shown, Reward-Zero aims to eliminate hand-crafted rewards by relying only on raw language embedding, enhancing generalization across diverse tasks. Zero here signifies the absence of hand-crafted rewards, without explicit reward engineering. This is the Zero step toward more general, adaptable, and scalable RL that can learn from natural language descriptions and raw observations, much like humans do.

Figure 2: Example tasks and keyframes from the completion-sense mini benchmark. Each episode contains 2–4 annotated keyframes at known completion percentages (0%, 33%, 50%, 66%, 100%) extracted from successful ManiSkill Gu et al. (2023) rollouts. The benchmark includes tasks with varying visual complexity, from large state changes (e.g., OpenCabinetDrawer) to fine-grained manipulations (e.g., PegInsertionSide).

Task	Frames	Completions (%)	Goal Description
OpenCabinetDrawer ( $\times$ 2)	4	0, 33, 66, 100	The cabinet drawer is fully open
AnymalC-Reach	4	0, 33, 66, 100	The quadruped robot is at the target position
PushCube	4	0, 33, 66, 100	The cube is at the target position
PegInsertionSide	4	0, 33, 66, 100	The peg is fully inserted into the hole
StackCube	4	0, 33, 66, 100	The red cube is stacked on the green cube

Table 1: Tasks in the completion-sense mini benchmark. Each episode contains keyframes at known completion percentages extracted from successful ManiSkill rollouts. Two episodes are included for OpenCabinetDrawer to test consistency across different initial configurations.

VLM-caption pipeline (Qwen2.5-VL-3B + MiniLM)
Method	FTA ( $\uparrow$ )	J+ ( $\uparrow$ )	Mono ( $\uparrow$ )	Latency
+ progress-description	12/18	5/6	2/6	$\sim$ 2 s
+ observe-only	12/18	6/6	1/6	$\sim$ 2 s
+ evidence-gating	0/18	0/6	0/6	$\sim$ 2 s
CLIP-direct (ViT-B/32), $\alpha=0.7$
with baseline penalty	13/18	6/6	2/6	$\sim$ 5 ms

Table 2: Mini benchmark results comparing VLM-caption and CLIP-direct approaches on 18 forward transitions across 6 episodes. FTA : forward transition accuracy (out of 18). J+ : jump detection, 0%

\rightarrow

100% (out of 6). Mono : fraction of episodes with fully monotonic potentials (out of 6). Latency is per-frame inference time on a single A30 GPU. Best results in bold . CLIP results are deterministic.

Figure 3: CLIP potential

\Phi(s)

vs. task completion (%) for each benchmark task (CLIP-direct,

\alpha=0.7

). OpenCabinetDrawer shows two episodes (solid/dashed) to illustrate consistency across initial configurations. All tasks now use four keyframes at 0%, 33%, 66%, and 100% completion. Tasks with large visual changes show strong monotonic trends; fine-manipulation tasks exhibit smaller potential ranges.

Figure 4: The AnymalC-Reach task involves a quadruped robot learning to navigate to a target location. The task requires the agent to understand spatial relationships and adapt its locomotion strategy accordingly. This environment serves as a challenging testbed for evaluating the effectiveness of our Reward-Zero in guiding learning through language-driven implicit rewards.

Panel of Figure 5 — Figure 5: Performance comparison of the ANYmal-C Reach-PPO baseline versus our approach. Solid lines represent the mean values over $2\text{M}$ training steps. Our method significantly outperforms the baseline in both task success rates and cumulative reward.

Panel of Figure 6 — Figure 6: Training dynamics comparison between the PPO baseline (orange) and Ours (blue). The Reward-Zero variant exhibits substantially smoother value loss, more stable policy updates, and more consistent KL and clipping behavior, indicating improved critic reliability and better - controlled policy optimization. Reward-Zero also shows faster entropy reduction and more stable explained variance, reflecting higher learning efficiency and more reliable advantage estimation throughout training.

Panel of Figure 7 — Figure 7: Ablation study on the scale parameter for Reward-Zero policy learning.

Panel of Figure 8 — Figure 8: Ablation on Reward - Zero Invocation Frequency.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Keyframes from the six benchmark episodes (five task types) at four completion stages (0%, 33%, 66%, 100%). Each row shows one episode; columns correspond to the annotated completion percentages. Tasks range from visually salient changes (top: drawer opening, robot locomotion) to fine-grained manipulation where task-relevant objects are small relative to the scene (bottom: peg insertion, cube stacking).

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 250 papers

ROOT: Discovering Rewards for User-Specified Embodied Behaviors
Oct 3, 2026Eren Sadikoglu, Aditya Taparia, Xinyuan Liu +1Reward Design for RL Robotic RL
Robotics and Autonomous Systems, Arizona State University, Tempe, Arizona, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA
Prompt-Driven Exploration
Jul 9, 2026Sunshine Jiang, John Marangola, David Zhang +6LLM-Guided RL RL Exploration
Massachusetts Institute of Technology · Improbable AI Lab · MIT-IBM Computing Research Lab
Specifying Reward Functions for RL Without Environment Sampling
Sep 14, 2026Stephane Hatgis-Kessell, W. Bradley Knox, Emma BrunskillLLM-Guided RL Reward Design for RL
1Computer Science Department, Stanford University · 2Computer Science Department, The University of Texas at Austin

Oct 3, 2026J/K move · Enter open · S save

ROOT: Discovering Rewards for User-Specified Embodied Behaviors

Eren Sadikoglu, Aditya Taparia, Xinyuan Liu, Ransalu Senanayake

Robotics and Autonomous Systems, Arizona State University, Tempe, Arizona, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA

Reward Design for RL Robotic RL LLM-Guided RL RL for Legged Locomotion

Open paper AskPDF arXiv

Reinforcement learning for embodied control remains constrained by the difficulty of reward specification. Although recent large language model (LLM)-based methods can synthesize reward functions from natural-language descriptions, they often fail to capture subtle behavioral properties that humans care about, such as natural gait, posture, and movement style. This limitation arises because many desired behaviors are easier to recognize visually than to encode in a reward function. We introduce Reward Optimization via Observable Trees (ROOT), a framework for discovering reward functions that align learned policies with user-specified embodied behaviors. Rather than relying solely on scalar training statistics, ROOT casts reward design as an observation-guided search over a persistent experiment tree that stores reward programs, trained policies, and rollout observations, together with behavioral insights distilled by a video-language model, to diagnose behavioral failures and guide subsequent reward refinements. We evaluate ROOT on seven tasks across four embodiments: simulated Hopper, HalfCheetah, Ant, Unitree Go2, and as well as the real-world Unitree Go2. ROOT produces behaviors that better align with user intent than those generated by existing LLM-based reward-generation methods, achieving up to 86.8% locomotion-completeness accuracy and improving Vid-LLM behavioral alignment from 3.56/5 to 4.14/5, a 16.5% improvement over baselines. Human evaluations further support these results, with ROOT preferred in 51-63% of pairwise comparisons.

Abstract

Reward Design for RL

Explore similar work

ROOT: Discovering Rewards for User-Specified Embodied Behaviors

Prompt-Driven Exploration

Specifying Reward Functions for RL Without Environment Sampling