Reward function design remains a bottleneck in reinforcement learning. While large language models (LLMs) have enabled automated reward generation, existing methods generate and revise reward functions as monolithic programs, making it difficult to reliably preserve and reuse effective components discovered in earlier iterations, leading to unstable performance across iterations. To address this, we propose Module Level Reward Evolution Framework (MLREF). At the core of MLREF is a module pool, a persistent repository of reusable reward components. MLREF treats the module pool as the primary optimization object: the pool evolves across iterations by accumulating successful modules, refining underperforming ones, and reusing proven components; while reward functions are constructed as linear combinations of modules drawn from this pool. To drive this evolution, MLREF integrates three mechanisms: reflection-based refinement, hybrid credit assignment, and a merge strategy with rollback, which together improve the effectiveness and robustness of reward optimization. Experiments on 17 tasks show that MLREF outperforms strong baselines by 25.2% in locomotion and 6.6% in manipulation, with more stable optimization dynamics.
Figures & tables
Property
EUREKA
RF-Agent
MLREF (Ours)
Optimization Unit
Function
Function
Module Pool
Module Identity
–
Anonymous
Named + Spec.
Cross-iter. Persistence
–
Partial
Persistent Pool
Credit Assignment
Holistic
Holistic
Per-Module Hybrid
Failure Recovery
–
Implicit
Explicit Rollback
Semantic Guidance
–
–
Task + Feedback Refl.
Table 1: Comparison of key design properties across LLM-based reward design methods. MLREF is the only framework that treats modules as first-class, persistently managed objects with per-module credit assignment, semantic guidance, and a principled failure recovery mechanism.
Figure 1: Overview of the MLREF pipeline. In the initialization phase (steps 1–3), the LLM performs task and environment reflection and generates diverse module pool variants. In the iterative optimization phase (steps 4–9), each variant assembles a reward function via weighted linear combination for RL evaluation; a merge strategy with rollback consolidates successful modules; and feedback reflection guides the next round of pool improvement.
Task
EUREKA
RF-Agent
MLREF (Ours)
Locomotion
Allegro Hand
25.48 ± 1.27
27.41 ± 1.90
28.89 ± 1.36
Ant
12.35 ± 1.29
12.34 ± 0.70
12.47 ± 0.30
Anymal
-0.0057 ± 0.0021
-0.0064 ± 0.0010
-0.0073 ± 0.0019
Franka Cabinet
0.397 ± 0.202
0.701 ± 0.200
0.997 ± 0.006
Humanoid
7.95 ± 0.87
5.56 ± 1.63
7.40 ± 1.12
Table 2: Performance comparison across 17 tasks. Each row shows the mean and standard deviation of the raw performance metric. The score of locomotion tasks is normalized before averaging. The percentage improvement is relative to the best baseline. Bold indicates the best mean result.
Method
Ant
Block Stack
Bottle Cap
Avg. % of Full
Full MLREF
12.47 ± 0.30
0.313 ± 0.244
0.987 ± 0.012
100.0%
GPT-4o
10.77 ± 1.95
0.101 ± 0.042
0.041 ± 0.027
40.9%
No Pool
11.51 ± 0.64
0.085 ± 0.040
0.988 ± 0.010
73.2%
No Reflection
9.58 ± 0.55
0.057 ± 0.032
0.639 ± 0.401
53.3%
No Weight Opt.
11.68 ± 1.14
0.044 ± 0.025
0.829 ± 0.338
63.9%
Table 3: Ablation results on three representative tasks. Best results are bold.
Figure 2: Performance evolution across iterations (best sample per iteration). Left: all-task average. Middle: Shadow Hand. Right: Catch Abreast. Shaded areas denote standard deviation.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
MLREF — Optimization
Max iterations I
9
Parallel samples per iteration S
3
Pipeline runs
3
RL evaluation seeds
5
MLREF — Pool Merge
Appendix
Table 4: Hyperparameters for MLREF and baseline methods.
Figure 3: Evolution curves for all 7 locomotion tasks.
Figure 4: Evolution curves for all 10 manipulation tasks.
Figure 5: Average evolution curves. Left: all 17 tasks. Middle: locomotion (7 tasks). Right: manipulation (10 tasks).
Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging and labor-intensive process due to the inefficiencies and inconsistencies inherent in traditional methods. Existing methods often rely on extensive manual design and evaluation steps, which are prone to redundancy and overlook local uncertainties at intermediate decision points. To address these challenges, we propose the Chain of Uncertain Rewards (CoUR), a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation in RL environments. Specifically, our CoUR introduces code uncertainty quantification with a similarity selection mechanism that combines textual and semantic analyses to identify and reuse the most relevant reward function components. By reducing redundant evaluations and leveraging Bayesian optimization on decoupled reward terms, CoUR enables a more efficient and robust search for optimal reward feedback. We comprehensively evaluate CoUR across nine original environments from IsaacGym and all 20 tasks from the Bidexterous Manipulation benchmark. The experimental results demonstrate that CoUR not only achieves better performance but also significantly lowers the cost of reward evaluations.
Large language model alignment via reinforcement learning depends critically on reward function quality. However, static, domain-specific reward models are often costly to train and exhibit poor generalization in out-of-distribution scenarios encountered during RL iterations. We present RLAR (Reinforcement Learning from Agent Rewards), an agent-driven framework that dynamically assigns tailored reward functions to individual queries. Specifically, RLAR transforms reward acquisition into a dynamic tool synthesis and invocation task. It leverages LLM agents to autonomously retrieve optimal reward models from the Internet and synthesize programmatic verifiers through code generation. This allows the reward system to self-evolve with the shifting data distributions during training. Experimental results demonstrate that RLAR yields consistent performance gains ranging from 10 to 60 across mathematics, coding, translation, and dialogue tasks. On RewardBench-V2, RLAR significantly outperforms static baselines and approaches the performance upper bound, demonstrating superior generalization through dynamic reward orchestration. The data and code are available on this link: https://github.com/ZhuoerFeng/RLAR.
Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at https://github.com/deng-ai-lab/PRM4RL.