Dual-Evolving Reinforcement Learning for Automated Reward Design

Sitao Cheng; Tianle Li; Xuhan Huang; Xunjian Yin; Difan Zou

cs.AIDec 15, 2025

Dual-Evolving Reinforcement Learning for Automated Reward Design

Authors: Sitao Cheng, Tianle Li, Xuhan Huang, Xunjian Yin, Difan Zou

Organizations: University of Waterloo · The University of Hong Kong · The Chinese University of Hong Kong, Shenzhen · Duke University

Abstract

Crafting effective reward signals remains a central challenge in reinforcement learning (RL) for complex reasoning. Existing methods for automated reward optimization search with a fixed proposer which never learns from its costly trials. We present Dual-Evolving Reinforcement Learning (DERL), a dual-training framework for autonomous discovery of reward structures. An outer-loop proposer composes a reward from task primitives, and an inner policy trains with this reward. The proposer itself learns by policy gradient from the validation performance of the inner policy. Thus, DERL utilizes past-trial feedback to shape its reward-proposal distribution, an empirical meta-gradient of task success. We validate DERL on embodied (ALFWorld), scientific (ScienceWorld), and mathematical (GSM8K, MATH) reasoning. On multi-turn agent benchmarks, DERL outperforms strong baselines, significantly on out-of-distribution tasks. In mathematics, DERL improves on the outcome reward where directly added primitives hurt. Further analyses under matched compute show that our proposer effectively captures the meta-gradient to discover useful reward signals.

Figures & tables

Figure 1: Left : Three reward design frameworks, from most to least human labor. Early work adopts heuristics or a trained reward model. A static proposer generates and refines rewards with a frozen LLM. In DERL , the proposer is itself optimized by policy gradient on the inner policy’s validation performance, a dual evolution of both the policy and the reward proposer. Right : Test score (%) with different rewards.

Figure 2: Our dual-evolving framework ( DERL ). The proposer $\psi$ reads a fixed task instruction and samples $n$ reward programs $\Phi=\{\phi_{1},\dots,\phi_{n}\}$ . Each program trains one inner policy, whose validation score $v_{i}$ is the reward of $\phi_{i}$ , and the group-normalized advantages update $\psi$ with GRPO. The inner-loop trains policy $\theta_{i}$ with GRPO under the generated reward $R_{\phi_{i}}$ (the index $n$ inside this block denotes the inner group size $G$ of Equation 5 ).

Method	ALFWorld			ScienceWorld
Method	L0	L1	L2 (OOD)	L0	L1	L2 (OOD)
GRPO w/ Outcome	76.6	71.1	29.7	21.1	13.7	10.9
GRPO w/ Average(primitives)	88.1	85.4	30.5	37.9	31.3	18.0
GiGPO	86.7	83.2	48.0	25.8	15.2	4.7
RLVMR	89.1	87.9	56.3	46.9	34.4	26.5
DERL -base	91.0	89.1	65.0	47.7	43.0	30.1

Table 1: Success rate (%) on ALFWorld and ScienceWorld with Qwen2.5-1.5B-Instruct at three levels as the mean over four seeds (statistics in Appendix G ). GiGPO and RLVMR results are from Zhang et al. (2025b) . Bold : best. Underline : second best. DERL -base beats all baselines at every level, significantly on OOD tasks.

Reward Function	GSM8K	MATH
Trained on MATH
Outcome	82.9	59.1
Outcome + Format	83.9	56.8
Average(primitives)	83.6	54.9
DERL -base	83.2	60.5
DERL -full	84.1	60.9

Table 2: Exact match (%) on GSM8K and MATH with Qwen2.5-3B . Bold : best. Underline : second best.

Figure 3: Measn GSM8K test exact match (%) of the inner policies ( Qwen2.5-1.5B ) trained with the proposals of each outer step. RLAIF stays flat. Random search fluctuates and ends lower. only DERL improves, because its proposer learns.

Figure 4: Outer-loop dynamics on ALFWorld (L2), GSM8K, and MATH. The x-axis is the outer step. Each point is the mean over the

n

inner policies trained from that step’s proposals, scored on the validation split (blue, the outer-loop reward) and on the test set (orange, never used for training). Dashed lines are linear trends.

Figure 5: GSM8K exact match (%) when a primitive is removed or flipped ( Qwen2.5-1.5B inner policy).

Figure 6: Share of stable and unstable structures among eight programs proposed at each outer step of an ALFWorld L2 run (nine steps, dynamics in Appendix E ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Success rate over 100 training epochs on ScienceWorld L0 for DERL -full (Three rounds of 33 epochs. Each round stars from the best inner-policy of the previous one) and GRPO w/ Average(primitives).

Figure 8: Outer-loop dynamics of the ALFWorld L2 run analyzed in Figure 6 , which converges in nine steps. Axes and curves are as in Figure 4 .

Outer step	Proposed program (verbatim)	Val score
0	g1 * (g2 - 1) / 2 + (g3 + 1) * (g4 - 1) * 2 / 3	0
	g1 + 0.5 * (g2 + 0.5 * (g3 + 0.5 * (g4)))) - 0.5 * (g2 + 0.5 * (g3 + ...	0
	g1 + 0.01 * (g2 - 0.001) + 0.0001 * (g3 - 0.0001) + 0.000005 * (g4 - ...	0.850
	g1 * 0.5 + 0.2 * (g2 + 0.1) - 0.3 / 2 + 0.4 * (g3 * 0.1) + 0.1 * (g4 * ...	0.879
	g1 * (g2 + (g3 - 1.0) * (g4 - 0.0))	0.023
	g1 + 0.5 * (g2 + 0.5 * (g3 + 0.5 * (g4 + 0.5)))	0.885

Appendix

Table 3: Programs proposed in the first four outer steps of one ALFWorld L2 run, with the validation performance of the inner policy trained with each. Rows with ... are cut for display. The text explains the two sources of a 0 score.

GRPO baselines
Method	Train	Test
GRPO w/ Outcome	89.45	75.82
GRPO w/ Average(primitives)	89.74	76.12
Static proposers
Random search (best over steps)	89.59	76.50
RLAIF (prompted GPT-4o)	89.23	77.18

Appendix

Table 4: Exact match (%) of the final policy on GSM8K in the studies of Appendices F.1 – F.4 . Bold : best.

Outer step	DERL -base	Random search	RLAIF
0	75.13	76.08	76.88
1	76.60	74.90	76.67
2	77.69	75.97	76.40
3	77.13	54.58	76.85
4	77.29	76.34	76.45
5	78.28	75.16	76.17

Appendix

Table 5: Mean GSM8K test accuracy (%) of the inner policies trained from the proposals at each outer step. Only the trained proposer improves over steps.

Method	L0	L1	L2
ALFWorld
GRPO w/ Outcome	76.56 $\pm$ 1.91 [73.52, 79.61]	71.09 $\pm$ 2.89 [66.50, 75.69]	29.69 $\pm$ 2.12 [26.32, 33.05]
GRPO w/ Average(primitives)	88.09 $\pm$ 1.33 [85.96, 90.21]	85.35 $\pm$ 1.33 [83.23, 87.47]	30.47 $\pm$ 2.63 [26.28, 34.65]
DERL -base	91.02 $\pm$ 2.07 [87.73, 94.30]	89.06 $\pm$ 1.42 [86.80, 91.32]	65.04 $\pm$ 2.64 [60.83, 69.25]
DERL -full	91.80 $\pm$ 1.35 [89.64, 93.95]	88.28 $\pm$ 1.78 [85.46, 91.11]	76.37 $\pm$ 4.05 [69.92, 82.82]
ScienceWorld

Appendix

Table 6: Success rate (%) on ALFWorld and ScienceWorld: mean

\pm

standard deviation and 95% confidence interval.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 259 papers

$R^2E:$ A Role-driven Reward Evolutionary Framework for Automated Reward Function Design
Oct 23, 2026Shouhao Chang, Xuan Liu, Hongye Zhu +2Reward Design for RL LLM-Guided RL
Hunan University · Central South University
Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning
Apr 15, 2026Shentong MoReward Design for RL LLM-Guided RL
Carnegie Mellon University
Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning
Jun 18, 2026Alan Nadelsticher RuvalcabaReinforcement Learning Sparse-Reward RL
College of Computing at the Georgia Institute of Technology, Georgia, USA · School of Engineering and Applied Science at the University of Pennsylvania, Pennsylvania, USA

Oct 23, 2026J/K move · Enter open · S save

$R^2E:$ A Role-driven Reward Evolutionary Framework for Automated Reward Function Design

Shouhao Chang, Xuan Liu, Hongye Zhu, Xinning Chen+1

Hunan University · Central South University

Reward Design for RL LLM-Guided RL Evolutionary RL Robotic RL

Open paper AskPDF arXiv

Designing effective reward functions is a critical challenge in Reinforcement Learning (RL), which traditionally requires costly manual trial-and-error. While Large Language Model (LLM)-based methods have shown promise in reward design, undifferentiated sampling and limited use of historical feedback often lead to insufficient exploration of the reward space and optimization instability. To address these challenges, we propose

R^2E

, a Role-driven Reward Evolutionary framework for automated reward function design that structures the process through explicit role specialization.

R^2E

employs multiple LLM roles with complementary objectives: an Explorer that promotes novelty to expand the search space, a Guardian that performs conservative refinements to improve training stability, and an Evolver that recombines high-performing rewards via crossover and mutation to integrate effective structures. A dynamic scheduling strategy coordinates these roles, progressively shifting the search from broad exploration to focused exploitation. To mitigate optimization instability and provide a medium for cross-role collaboration,

R^2E

incorporates a Global Elite Pool that retains the best-performing rewards to guide subsequent generations, thereby enabling coordinated interaction among roles and ensuring a reliable and consistent refinement process. Extensive experiments on multiple robotic tasks demonstrate the effectiveness of our method.

Abstract

Reward Design for RL

Explore similar work

R2E:R^2E:R2E: A Role-driven Reward Evolutionary Framework for Automated Reward Function Design

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning

Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning

$R^2E:$ A Role-driven Reward Evolutionary Framework for Automated Reward Function Design