Decoupling Task and Behavior: A Two-Stage Reward Curriculum in Reinforcement Learning for Robotics

Kilian Freitag; Knut Åkesson; Morteza Haghir Chehreghani

cs.LGMar 5, 2026

Decoupling Task and Behavior: A Two-Stage Reward Curriculum in Reinforcement Learning for Robotics

Authors: Kilian Freitag, Knut Åkesson, Morteza Haghir Chehreghani

Organizations: Chalmers University of Technology · Gothenburg University

Abstract

Deep Reinforcement Learning is a promising tool for robotic control, yet practical application is often hindered by the difficulty of designing effective reward functions. Real-world tasks typically require optimizing multiple objectives simultaneously, necessitating precise tuning of their weights to learn a policy with the desired characteristics. To address this, we propose a two-stage reward curriculum where we decouple task-specific objectives from behavioral terms. In our method, we first train the agent on a simplified task-only reward function to ensure effective exploration before introducing the full reward that includes auxiliary behavior-related terms such as energy efficiency. Further, we analyze various transition strategies and demonstrate that reusing samples between phases is critical for training stability. We validate our approach on the DeepMind Control Suite, ManiSkill3, and a mobile robot environment, modified to include auxiliary behavioral objectives. Our method proves to be simple yet effective, substantially outperforming baselines trained directly on the full reward while exhibiting higher robustness to specific reward weightings.

Figures & tables

Figure 1: Comparison of the baseline TD3 and SAC algorithms with reward curriculum versions (RC-TD3 and RC-SAC). The first row shows the achieved base reward (DM Control) or success rate (MobileRobot and ManiSkill3) and the second row shows the average reward. Both are computed as the average over the last 50k training steps and 3 random seed using target weight $w_{\text{target}}=0.5$ for DM Control, $w_{\text{target}}=0.25$ for ManiSkill3 and $w_{\text{target}}\in\{0.25,0.5,0.75\}$ for MobileRobot (MR).

Figure 2: Exemplary environments from DM control (walker-run), MobileRobot, and ManiSkill3 (PullCubeTool-v1).

Figure 3: Comparison of the mean success rate and average reward for ManiSkill3 (4 environments aggregated) and MobileRobot with different $w_{\text{target}}$ of RC-TD3 and RC-SAC to their baselines. The average reward is computed using a $w_{\text{target}}=0.5$ to make results comparable.

Figure 4: Comparison of the mean episode reward of RC-SAC with different switches as described in Section 3.1 . The values are average over the last 50 [k] training steps.

Figure 5: Comparison of the mean episode reward for different transitions as described in Section 3.2 . The values are average over the last 50 [k] training steps.

Figure 6: Comparison of RC-SAC with SAC and resetting the buffer or networks when switching between curriculum phases. Results are computed using 2 random seeds. Results are smoothed using a Savitzky-Golay filter with a window length 100 and polyorder 2.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Exemplary environment maps used for training. Obstacle positions, paths, initial states, and goal positions are randomized. While maps 0 and 2 contain dynamic obstacles, maps 1 and 3 only contain static ones.

Figure 8: Functions for dense reward terms with $\kappa=0.942$ , $v_{\text{ref}}=1.2$ and $d_{\text{track,max}}=5$ . The range for each term is normalized to $[-1,1]$ . Green shows the reward shaping term that enables finding the goal faster. The other penalizing terms are colored in red.

Figure 9: Different reward formulations for MobileRobot for 3 random seeds. Results are smoothed using a Savitzky-Golay filter with window length 200 and polyorder 2.

Method	TD3	RC-TD3	SAC	RC-SAC
Environment
finger-turn-easy	0.56 $\pm$ 0.05	0.49 $\pm$ 0.02	0.71 $\pm$ 0.04	0.65 $\pm$ 0.05
reacher-hard	0.76 $\pm$ 0.13	0.82 $\pm$ 0.15	0.90 $\pm$ 0.01	0.91 $\pm$ 0.01
reacher-easy	0.91 $\pm$ 0.02	0.92 $\pm$ 0.01	0.92 $\pm$ 0.01	0.91 $\pm$ 0.02
pendulum-swingup	0.49 $\pm$ 0.01	0.53 $\pm$ 0.04	0.49 $\pm$ 0.01	0.56 $\pm$ 0.06
fish-upright	0.76 $\pm$ 0.03	0.79 $\pm$ 0.07	0.77 $\pm$ 0.03	0.77 $\pm$ 0.06

Appendix

Table 1: Mean and standard deviation of the average reward for SAC and RC-SAC for all different environments over 3 random seeds average over the last 50 [k] training steps. Here we use

w_{\text{target}}=0.5

for DM Control envs,

w_{\text{target}}=0.25

for ManiSkill3 and

w_{\text{target}}=0.75

for MobileRobot.

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 252 papers

Stage-Transition Dense Reward Modeling for Reinforcement Learning
Jun 30, 2026Yang Yang, Bingjie Chen, Zihan Wang +4RL for Robotics Long-Horizon Robotic Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China. · Zerith Robotics, Shenzhen 518055, China.
Enhancing Deep Reinforcement Learning for Robotic Motion Planning via Task Decomposition
Oct 26, 2026Yuanzhi He, Victor Romero-Cano, Jiayu Xu +2Reinforcement Learning Robotic RL
School of Computational and Mathematical Sciences, Cardiff University, United Kingdom
Disentangling Task-Relevant Latent Dynamics for Reward Modeling
Oct 9, 2026Dayou Li, Jionghao Wang, Peihao Wang +9Dense Reward RL Multimodal Reward Modeling
Texas A&M University · University of Texas at Austin · Google DeepMind +1

Jun 30, 2026J/K move · Enter open · S save

Stage-Transition Dense Reward Modeling for Reinforcement Learning

Yang Yang, Bingjie Chen, Zihan Wang, Yizhe Li+3

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China. · Zerith Robotics, Shenzhen 518055, China.

RL for Robotics Long-Horizon Robotic Manipulation Dense Reward RL Visual RL Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning for long-horizon robotic manipulation is often limited by sparse and delayed rewards, while manually designing dense shaping signals is costly and brittle to changes in environments and object configurations. This work proposes Stage-Transition Dense Reward (STDR), a visual reward-learning framework that converts unstructured expert videos into logically grounded dense rewards for training RL agents from scratch. STDR leverages semantic understanding to infer a task's stage structure from demonstrations, and delivers two complementary learning signals during online training: (i) stage-transition feedback that provides goal-directed reward, and (ii) within-stage progress feedback that supplies fine-grained guidance toward completing each stage. Furthermore, an out-of-distribution (OOD) detection mechanism and a grasping regulation module are integrated to enhance robustness and prevent reward hacking. Experiments on 14 manipulation tasks across MetaWorld, ManiSkill, and Franka Kitchen show that STDR consistently improves sample efficiency and success rates over multiple baselines, and matches or surpasses handcrafted dense rewards on several challenging tasks. Real-robot evaluations further indicate that STDR assigns stable, progress-aligned rewards on successful executions while producing appropriately low rewards for failures, suggesting robustness to visual noise and better-calibrated reward assignment across settings.

Abstract

RL for Robotics

Explore similar work

Stage-Transition Dense Reward Modeling for Reinforcement Learning

Enhancing Deep Reinforcement Learning for Robotic Motion Planning via Task Decomposition

Disentangling Task-Relevant Latent Dynamics for Reward Modeling