$R^2E:$ A Role-driven Reward Evolutionary Framework for Automated Reward Function Design

Shouhao Chang; Xuan Liu; Hongye Zhu; Xinning Chen; Shigeng Zhang

cs.ROOct 23, 2026

$R^2E:$ A Role-driven Reward Evolutionary Framework for Automated Reward Function Design

Authors: Shouhao Chang, Xuan Liu, Hongye Zhu, Xinning Chen, Shigeng Zhang

Organizations: Hunan University · Central South University

Abstract

Designing effective reward functions is a critical challenge in Reinforcement Learning (RL), which traditionally requires costly manual trial-and-error. While Large Language Model (LLM)-based methods have shown promise in reward design, undifferentiated sampling and limited use of historical feedback often lead to insufficient exploration of the reward space and optimization instability. To address these challenges, we propose $R^2E$ , a Role-driven Reward Evolutionary framework for automated reward function design that structures the process through explicit role specialization. $R^2E$ employs multiple LLM roles with complementary objectives: an Explorer that promotes novelty to expand the search space, a Guardian that performs conservative refinements to improve training stability, and an Evolver that recombines high-performing rewards via crossover and mutation to integrate effective structures. A dynamic scheduling strategy coordinates these roles, progressively shifting the search from broad exploration to focused exploitation. To mitigate optimization instability and provide a medium for cross-role collaboration, $R^2E$ incorporates a Global Elite Pool that retains the best-performing rewards to guide subsequent generations, thereby enabling coordinated interaction among roles and ensuring a reliable and consistent refinement process. Extensive experiments on multiple robotic tasks demonstrate the effectiveness of our method.

Figures & tables

Figure 1: Comparison of reward design paradigms. (Eureka) generates homogeneous rewards and lacks memory. If iteration $T+1$ regresses, the LLM references only $T+1$ for $T+2$ , discarding high-performing rewards from $T$ . (Our Method) populates a Global Elite Pool via initial exploration. Specialized roles query historical elites from the pool to iteratively evolve diverse reward candidates.

Figure 2: Overall architecture of R 2 E. The framework operates in three main stages: (Stage 1 Initial Iteration) : Reward candidates are generated through a broad exploration phase based on the environment code and task descriptions, and then trained on IsaacGym or Bi-DexHands tasks to initialize the Global Elite Pool with the Top 5 functions. (Stage 2 Dynamic Role-driven Iterations) : In each iteration, a total of $N=16$ candidates are generated through dynamic scheduling by the Explorer, Guardian, and Evolver roles. After RL training and evaluation, the pool is dynamically updated by replacing the worst-performing reward. (Stage 3 End of Iterations) : After the search concludes, the Top 1 reward is selected and re-evaluated across five random seeds to produce the final output. (Global Elite Pool) : A fixed-capacity memory ( $K=5$ ) that stores and ranks the best-performing reward functions across iterations.

Method	Ant	Anymal	Quadcopter	AllegroHand	FrankaCabinet	ShadowHand	Avg Norm Score
Sparse	1.0000 $\pm$ 0.0000	-1.9619 $\pm$ 0.2734	-0.3447 $\pm$ 0.2361	0.0619 $\pm$ 0.0415	0.0000 $\pm$ 0.0001	0.0546 $\pm$ 0.0046	0.0000
Human	6.3275 $\pm$ 0.3159	-0.0232 $\pm$ 0.0030	-0.0906 $\pm$ 0.0095	11.3698 $\pm$ 0.8865	0.1230 $\pm$ 0.0892	6.9433 $\pm$ 1.3385	1.0000
Eureka	6.5876 $\pm$ 0.2602	-0.0249 $\pm$ 0.0019	-0.0415 $\pm$ 0.0114	19.2377 $\pm$ 3.0125	0.3816 $\pm$ 0.2243	4.9272 $\pm$ 2.3201	1.4578
REvolve	4.4297 $\pm$ 1.1706	-0.0172 $\pm$ 0.0021	-0.0388 $\pm$ 0.0135	21.9641 $\pm$ 1.3215	0.7224 $\pm$ 0.152	8.1640 $\pm$ 1.6612	1.8500
RF-Agent	6.7225 $\pm$ 0.2767	-0.0154 $\pm$ 0.0053	-0.0274 $\pm$ 0.0028	20.4162 $\pm$ 2.1523	0.8156 $\pm$ 0.1571	7.9717 $\pm$ 3.5775	2.1512
Ours	7.2639 $\pm$ 0.4584	-0.0135 $\pm$ 0.0043	-0.0211 $\pm$ 0.0076	23.4458 $\pm$ 0.7224	1.0000 $\pm$ 0.0000	8.9603 $\pm$ 1.4171	2.4908

Table 1: Performance Comparison on IsaacGym Tasks. All results report task-specific performance metrics as mean

\pm

standard deviation across runs. Bold indicates the best results within each group. Avg norm score denotes the average human-normalized performance computed as

(\text{Method}-\text{Sparse})/(\text{Human}-\text{Sparse})

for each task.

Figure 3: Performance Comparison on Bi-DexHands Tasks. Bars and error bars denote the mean and standard deviation of success rates over 5 random seeds, respectively. R 2 E achieves the highest mean success rate and maintains competitive stability across all environments.

Figure 4: Training curves on representative locomotion and manipulation tasks from IsaacGym and Bi-DexHands. Solid lines and shaded regions denote the mean and standard deviation over 5 seeds. IsaacGym curves report task-specific performance metrics, and Bi-DexHands curves report success rate.

Scheduling Strategy	Ant	FrankaCabinet	AllegroHand	Pen
Fixed (5:5:6 Allocation)	$6.5833\pm 0.3008$	$0.8157\pm 0.0206$	$20.7247\pm 1.9726$	$0.7277\pm 0.1685$
Dynamic (Ours)	$\mathbf{7.2639\pm 0.4584}$	$\mathbf{1.0000\pm 0.0000}$	$\mathbf{23.4458\pm 0.7224}$	$\mathbf{0.8967\pm 0.0891}$

Table 2: Impact of Scheduling Strategy on task performance.

Task	$K=3$	$K=5$ (Ours)	$K=8$	$K=12$	$K=16$
Ant	$6.8632\pm 0.2850$	$\mathbf{7.2639\pm 0.4584}$	$6.7844\pm 0.8383$	$5.5527\pm 0.4833$	$3.8366\pm 3.0366$
FrankaCabinet	$0.8621\pm 0.1026$	$\mathbf{1.0000\pm 0.0000}$	$0.7425\pm 0.0156$	$0.7627\pm 0.1036$	$0.5163\pm 0.1673$
AllegroHand	$23.2724\pm 1.6273$	$\mathbf{23.4458\pm 0.7224}$	$22.7741\pm 1.7614$	$21.6571\pm 1.6136$	$18.1642\pm 2.6115$
Pen	$0.8201\pm 0.0617$	$\mathbf{0.8967\pm 0.0891}$	$0.8013\pm 0.0968$	$0.7314\pm 0.0916$	$0.5614\pm 0.2901$

Table 3: Ablation study on the size of the Global Elite Pool (

K

Iterations ( $T$ )	Total candidates ( $T\times N$ )	Ant	AllegroHand	Pen
1	16	$4.8317\pm 0.3143$	$16.5829\pm 1.6192$	$0.3873\pm 0.1354$
3	48	$5.7638\pm 0.1865$	$20.6347\pm 1.0519$	$0.6516\pm 0.1017$
5	80	$7.2639\pm 0.4584$	$23.4458\pm 0.7224$	$0.8967\pm 0.0891$
7	112	$\mathbf{7.6821\pm 0.6817}$	$\mathbf{24.0685\pm 0.4739}$	$\mathbf{0.9166\pm 0.0539}$

Table 4: Performance under different search horizons with

N=16

. Bold indicates the highest value in each task.

T=5

is the setting used in the main experiments.

Budget ( $N$ )	Total candidates ( $5\times N$ )	Ant	AllegroHand	Pen
4	20	$6.0789\pm 0.6078$	$19.0723\pm 0.4367$	$0.6049\pm 0.1033$
8	40	$6.5672\pm 1.1067$	$20.9287\pm 0.8713$	$0.7549\pm 0.0376$
16	80	$7.2639\pm 0.4584$	$23.4458\pm 0.7224$	$0.8967\pm 0.0891$
24	120	$\mathbf{7.9834\pm 0.2569}$	$\mathbf{24.4028\pm 0.3644}$	$\mathbf{0.9534\pm 0.0135}$

Table 5: Performance under different numbers of candidates in each iteration with

T=5

. Bold indicates the highest value in each task.

N=16

is the setting used in the main experiments.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of IsaacGym Tasks. The benchmark suite is categorized into locomotion tasks ( Ant , Anymal , Quadcopter ) and manipulation tasks ( AllegroHand , FrankaCabinet , ShadowHand ).

Task	Obs.	Act.	Task Description
Ant	60	8	Maximize forward velocity while maintaining stability.
Anymal	48	12	Track random linear and angular velocity commands.
Quadcopter	21	12	Navigate to target coordinates and maintain stable hover.
AllegroHand	88	16	Reorient object to target pose using a 16-DoF hand.
FrankaCabinet	23	9	Interact with and open a cabinet drawer.
ShadowHand	211	20	Reorient object using a 24-DoF anthropomorphic hand.

Appendix

Table 6: Specifications of IsaacGym Environments. Summary of observation (Obs.) and action (Act.) dimensions along with primary task objectives.

Figure 8: Visualization of Bi-DexHands Tasks. These tasks require synchronized control of two agents to solve high-dimensional manipulation problems, such as pouring water ( Kettle ) or collaborative swinging ( SwingCup ).

Task	Obs.	Act.	Task Description
CatchAbreast	422	52	This class corresponds to the Catch Abreast task. This environment consists of two shadow hands placed side by side in the same direction and an object that needs to be passed. Compared with the previous environment which is more like passing objects between the hands of two people, this environment is designed to simulate the two hands of the same person passing objects, so different catch techniques are also required and require more hand translation and rotation techniques
DoorCloseOutward	417	52	This class corresponds to the DoorCloseOutward task. This environment also require a closed door to be opened and the door can only be pushed inward or initially open outward, but because they can’t complete the task by simply pushing, which need to catch the handle by hand and then open or close it, so it is relatively difficult
Kettle	417	52	This class corresponds to the PourWater task. This environment involves two hands, a kettle, and a bucket, we need to hold the kettle with one hand and the bucket with the other hand, and pour the water from the kettle into the bucket. In the practice task in IsaacGym, we use many small balls to simulate the water
LiftUnderarm	417	52	This class corresponds to the LiftUnderarm task. This environment requires grasping the pot handle with two hands and lifting the pot to the designated position. This environment is designed to simulate the scene of lift in daily life and is a practical skill
Pen	417	52	This class corresponds to the Open Pen Cap task. This environment involves two hands and a pen, we need to use two hands to open the pen cap
PushBlock	428	52	This class corresponds to the PushBlock task. This environment involves two hands and two blocks, we need to use both hands to reach and push the block to the desired goal separately. This is a relatively simple task

Appendix

Table 7: Detailed Specifications of Bi-DexHands Environments. The table lists the high-dimensional observation (Obs.) and action (Act.) spaces alongside the specific coordination logic for each task.

IsaacGym Tasks		Bi-DexHands Tasks
Task Name	$N_{\text{envs}}$	Task Name	$N_{\text{envs}}$
AllegroHand	16,384	CatchAbreast	2,048
Quadcopter	8,192	DoorCloseOutward	2,048
ShadowHand	8,192	Kettle	2,048
Ant	4,096	LiftUnderarm	2,048
Anymal	4,096	Pen	2,048

Appendix

Table 8: Parallel Environment Configurations. The specific number of parallel simulation instances (

N_{\text{envs}}

) utilized for each task to maximize GPU throughput.

Figure 9: Evolution of the best task-specific performance metric on IsaacGym. The curves represent the highest task-specific performance metric achieved by the best reward discovered so far as the number of iterations increases. Our method (R 2 E) demonstrates faster convergence (e.g., FrankaCabinet ) and higher asymptotic performance (e.g., AllegroHand , ShadowHand ) compared to baselines.

Figure 10: Evolution of Maximum Success Rate on Bi-DexHands Tasks. Comparisons across 8 complex manipulation tasks show that R 2 E consistently locates superior reward functions earlier in the search process. Notably, in difficult tasks like Pen , R 2 E succeeds in finding functional rewards where baselines stagnate.

Task Name	Total Time (H:M:S)
AllegroHand	11:57:05
Ant	00:48:19
Anymal	01:46:43
FrankaCabinet	01:34:50
Quadcopter	00:46:03
ShadowHand	06:42:25

Appendix

Table 9: Empirical wall-clock time for representative tasks (Total 5-iteration search).

Task Name	Prompt Tokens	Completion Tokens	Total Tokens	Est. Cost (RMB)
AllegroHand	76,118	31,464	107,582	0.32
Ant	73,182	38,234	111,416	0.36
Anymal	69,530	48,141	117,671	0.42
FrankaCabinet	60,717	34,121	94,838	0.32
Quadcopter	56,514	35,221	91,735	0.32
ShadowHand	75,419	23,899	99,318	0.27

Appendix

Table 10: Token Usage and Economic Cost Analysis. The table details the cumulative prompt and completion tokens consumed over a full 5-iteration evolutionary search, along with the estimated total API cost in RMB.

Task	LLM Backbone	Eureka	R 2 E
Ant	Qwen3-Coder-Flash	3.8851 $\pm$ 0.0289	6.2114 $\pm$ 0.6671
	Qwen3-Max	6.5876 $\pm$ 0.2602	7.2639 $\pm$ 0.4584
	GPT-5.4	6.3583 $\pm$ 0.6967	7.2615 $\pm$ 0.5690
ShadowHand	Qwen3-Coder-Flash	0.1626 $\pm$ 0.0866	0.5577 $\pm$ 0.3162
	Qwen3-Max	4.9272 $\pm$ 1.4171	8.9603 $\pm$ 1.4174
	GPT-5.4	7.8988 $\pm$ 0.5859	9.6329 $\pm$ 1.3460

Appendix

Table 11: Impact of LLM Backbone on Reward Evolution. Performance is reported as mean

\pm

standard deviation over 5 seeds.

LLM Backbone	Ant	Pen
Qwen3.7-Max only	$7.5604\pm 0.6164$	$0.9678\pm 0.0057$
DeepSeek-v4-pro only	$7.6408\pm 0.1465$	$0.9468\pm 0.0416$
GPT-5.6-sol-max only	$7.9866\pm 0.3364$	$0.9769\pm 0.0014$
Mixed (alternating LLMs)	$7.7542\pm 0.2541$	$0.9538\pm 0.0048$

Appendix

Table 12: Single and mixed LLM backbones. Ant reports the task-specific performance metric, and Pen reports success rate. Results are mean

\pm

standard deviation.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 261 papers

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning
Apr 15, 2026Shentong MoReward Design for RL LLM-Guided RL
Carnegie Mellon University
MLREF: Efficient Module Reuse for Reward Design in Reinforcement Learning via Large Language Models
Aug 19, 2026Chenglin Liu, Xun Wang, Ruishuo Chen +2Reward Design for RL Robotic RL
Institute for Interdisciplinary Information Sciences, Tsinghua University
RF-Agent: Automated Reward Function Design via Language Agent Tree Search
Feb 27, 2026Ning Gao, Xiuhui Zhang, Xingyu Jiang +3Reward Design for RL RL Control
Beihang University, 37 Xueyuan Road, Haidian District, Beijing

Apr 15, 2026J/K move · Enter open · S save

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning

Shentong Mo

Carnegie Mellon University

Reward Design for RL LLM-Guided RL Robotic RL

Open paper AskPDF arXiv

Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging and labor-intensive process due to the inefficiencies and inconsistencies inherent in traditional methods. Existing methods often rely on extensive manual design and evaluation steps, which are prone to redundancy and overlook local uncertainties at intermediate decision points. To address these challenges, we propose the Chain of Uncertain Rewards (CoUR), a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation in RL environments. Specifically, our CoUR introduces code uncertainty quantification with a similarity selection mechanism that combines textual and semantic analyses to identify and reuse the most relevant reward function components. By reducing redundant evaluations and leveraging Bayesian optimization on decoupled reward terms, CoUR enables a more efficient and robust search for optimal reward feedback. We comprehensively evaluate CoUR across nine original environments from IsaacGym and all 20 tasks from the Bidexterous Manipulation benchmark. The experimental results demonstrate that CoUR not only achieves better performance but also significantly lowers the cost of reward evaluations.

Abstract

Reward Design for RL

Explore similar work

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning

MLREF: Efficient Module Reuse for Reward Design in Reinforcement Learning via Large Language Models

RF-Agent: Automated Reward Function Design via Language Agent Tree Search