Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging and labor-intensive process due to the inefficiencies and inconsistencies inherent in traditional methods. Existing methods often rely on extensive manual design and evaluation steps, which are prone to redundancy and overlook local uncertainties at intermediate decision points. To address these challenges, we propose the Chain of Uncertain Rewards (CoUR), a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation in RL environments. Specifically, our CoUR introduces code uncertainty quantification with a similarity selection mechanism that combines textual and semantic analyses to identify and reuse the most relevant reward function components. By reducing redundant evaluations and leveraging Bayesian optimization on decoupled reward terms, CoUR enables a more efficient and robust search for optimal reward feedback. We comprehensively evaluate CoUR across nine original environments from IsaacGym and all 20 tasks from the Bidexterous Manipulation benchmark. The experimental results demonstrate that CoUR not only achieves better performance but also significantly lowers the cost of reward evaluations.
Figures & tables
Figure 1: Challenges in LLM-driven reward generation. While recent methods leverage Large Language Models (LLMs) to automate reward design, they typically treat the reward function as a monolithic entity optimized through inefficient trial-and-error. This approach fails to address local uncertainties at intermediate decision points, leading to redundant efforts, suboptimal policies, and high computational costs. Our proposed CoUR framework tackles these challenges by quantifying code uncertainty and decoupling reward terms for independent optimization.
Figure 2: Overview of the Chain of Uncertain Rewards ( CoUR ) framework. The pipeline consists of three main stages. Left: Given a natural language task description, a Large Language Model (LLM) generates an initial, multi-term reward function. Middle: Code Uncertainty Quantification (CUQ) analyzes the textual and semantic similarities of the generated reward components to compute uncertainty scores ( U -scores), identifying and refining ambiguous terms. Right: Bayesian Decoupling Optimization (BDO) decouples the reward terms and independently optimizes their hyperparameters via Bayesian optimization. BDO leverages the U -scores to prioritize exploration, ultimately recombining the optimized components into a robust final reward function.
Ri∗=argmaxRiJ(Ri).
Algorithm 1 Example Workflow for CoUR
Method
IsaacGym
Bidexterous
Human
1.00
45.90
Sparse
0.00
10.40
L2R [ 20 ]
1.69
24.40
Eureka [ 9 ]
2.66
55.30
Text2Reward [ 19 ]
2.78
56.87
CoUR (ours)
5.62
65.63
Table 1: Comparison results on IsaacGym and Bidexterous Manipulation tasks.
Reward function design remains a bottleneck in reinforcement learning. While large language models (LLMs) have enabled automated reward generation, existing methods generate and revise reward functions as monolithic programs, making it difficult to reliably preserve and reuse effective components discovered in earlier iterations, leading to unstable performance across iterations. To address this, we propose Module Level Reward Evolution Framework (MLREF). At the core of MLREF is a module pool, a persistent repository of reusable reward components. MLREF treats the module pool as the primary optimization object: the pool evolves across iterations by accumulating successful modules, refining underperforming ones, and reusing proven components; while reward functions are constructed as linear combinations of modules drawn from this pool. To drive this evolution, MLREF integrates three mechanisms: reflection-based refinement, hybrid credit assignment, and a merge strategy with rollback, which together improve the effectiveness and robustness of reward optimization. Experiments on 17 tasks show that MLREF outperforms strong baselines by 25.2% in locomotion and 6.6% in manipulation, with more stable optimization dynamics.
Chenglin Liu, Xun Wang, Ruishuo Chen +2
Institute for Interdisciplinary Information Sciences, Tsinghua University
Designing efficient reward functions for low-level control tasks is a challenging problem. Recent research aims to reduce reliance on expert experience by using Large Language Models (LLMs) with task information to generate dense reward functions. These methods typically rely on training results as feedback, iteratively generating new reward functions with greedy or evolutionary algorithms. However, they suffer from poor utilization of historical feedback and inefficient search, resulting in limited improvements in complex control tasks. To address this challenge, we propose RF-Agent, a framework that treats LLMs as language agents and frames reward function design as a sequential decision-making process, enhancing optimization through better contextual reasoning. RF-Agent integrates Monte Carlo Tree Search (MCTS) to manage the reward design and optimization process, leveraging the multi-stage contextual reasoning ability of LLMs. This approach better utilizes historical information and improves search efficiency to identify promising reward functions. Outstanding experimental results in 17 diverse low-level control tasks demonstrate the effectiveness of our method. The source code is available at https://github.com/deng-ai-lab/RF-Agent.
Designing effective reward functions in RL typically requires substantial expertise and trial and error. Recent work automates this process with LLM-based systems that generate and iteratively improve reward code using policy feedback. However, these methods are often hard to compare because they differ in implementation details, feedback assumptions, and evaluation environments. To address this, we introduce a Benchmark for Iterative Reward Design (BIRD) that expresses existing methods in a unified configuration and evaluation space. This lets us compare algorithms directly, ablate individual design choices, and prototype new components under matched feedback conditions and policy-training budgets. Across MuJoCo, Meta-World, Assistax, and HumanoidBench, we identify a small set of simple design choices that consistently improve performance. Combining these choices yields significantly better performance than the evaluated methods from prior work. Our results highlight the strength of simple baselines and motivate further study of when additional algorithmic complexity improves iterative reward design. Code is available at https://github.com/safety-research/bird.
Logan Mondal Bhamidipaty, Lauren Robson, Linda Petrini +2
Anthropic Fellows Program · University of Edinburgh · Anthropic