Organizations: Beijing Institute of Technology, Beijing, China · Heriot-Watt University, Edinburgh, U.K. · Shenzhen Institutes of Advanced Technology, CAS, Shenzhen, China · Microsoft Research NY, USA · Tsinghua University, Beijing, China · Jilin University, Changchun, China
Model-based reinforcement learning (MBRL) is sample-efficient but struggles in sparse reward settings. A critical bottleneck arises from the lack of informative gradients in sparse settings, where standard reward models often yield flat landscapes that struggle to guide planning. To address this challenge, we propose Shaping Landscapes with Optimistic Potential Estimates (SLOPE), a novel framework that shifts reward modeling from predicting sparse scalars to constructing informative potential landscapes. SLOPE employs optimistic distributional regression to estimate high-confidence upper bounds, which amplifies rare success signals and ensures sufficient exploration gradients. Evaluations on 30+ tasks across 5 benchmarks and real-world robotic deployments, demonstrate that SLOPE consistently outperforms leading baselines in fully sparse, semi-sparse, and dense rewards. The project code is available at https://github.com/Arya87/SLOPE.
Figures & tables
Figure 1: Key challenges in sparse-reward MBRL.
Method
Reward Shaping
Demonstrations
Learning Target
Reward Supervision
Dreamer-v3 Hafner et al. (2025)
✗
✗
Ground-truth reward
Sparse
TD-MPC2 Hansen et al. (2024)
✗
✗
Ground-truth reward
Sparse
MoDem Hansen et al. (2023)
✗
✓
Ground-truth reward
Sparse
DreamSmooth Lee et al. (2024)
✓
✗
Smoothed reward
Dense + Sparse
DEMO 3 Escoriza et al. (2025)
✓
✓
Stage-wise reward
Semi-sparse
SLOPE (Ours)
✓
✓
Potential-based reward
Sparse
Table 1: Comparison of MBRL algorithmic features under sparse-reward settings. We contrast the reward modeling paradigms and supervision types of SLOPE against major baselines, highlighting how each method transforms or utilizes sparse environmental signals.
Figure 2: Framework of SLOPE. Building upon MoDem’s multi-phase accelerated learning framework, we also introduce two training enhancements: (i) initializing MPPI sampling distribution from the prior policy πPrior , and (ii) augmenting the demonstration buffer with successful trajectories. The shaped reward r is used for training both the reward model Rθ and the Q value function Qθ . The environment input consists of multimodal observations o=(x,q) , where x denotes raw RGB images captured by the robot’s camera, and q represents the robot’s proprioceptive sensory information, e.g., the gripper state. Subsequent observations are encoded by the target encoder (illustrated in grey) which is updated via an exponential moving average (EMA) of the online encoder.
Figure 3: Toy example on a 10×10 GridWorld. Left : The original environment with sparse rewards ( +1 at goal, 0 elsewhere). Right : The dense reward signal generated by PBRS using converged optimal value function V∗ . ■ : wall.
Figure 4: Visualizations of representative tasks. Top : Simulation benchmarks including ManiSkill3 ( ms ), Meta-World ( mw ), Robosuite , and Adroit . Bottom : Real-world tasks including Press Button, Push Cube, and Grasp Cube.
Figure 5: Average success rates across 20 tasks from 4 benchmarks. Curves and shaded areas represent the mean and 95% confidence intervals (CIs) over 5 independent runs. We include both sparse and semi-sparse tasks for ManiSkill3 and Meta-World , while RoboSuite and Adroit involve sparse rewards only, following their native task designs. See Appendix C.3 for individual task details.
Method
Press Button
Push Cube
Grasp Cube
Success ( ↑ )
Intervention ( ↓ )
Success ( ↑ )
Intervention ( ↓ )
Success ( ↑ )
Intervention ( ↓ )
SAC
45.0
16.8
0.0
46.5
0.0
45.3
MoDem
65.0
10.0
15.0
18.4
35.0
24.0
DEMO 3
100.0
11.1
15.0
19.1
20.0
23.2
SLOPE (Ours)
100.0
7.3
65.0
18.2
65.0
21.3
Table 2: Comparison of methods on real-world tasks at 15 k environment steps ( ≈ 1 hour of real-world training). Results are averaged over 20 evaluation trials. Success Rate and Intervention Rate are reported in percentage (%). The best results are highlighted in colorbox .
Figure 6: Comparison with intrinsic-motivation and disagreement-based exploration baselines on sparse-reward ManiSkill3 tasks. All variants share the MoDem backbone and demonstrations. Compared with RND, Q-ensemble disagreement, and Plan2Explore-style disagreement, SLOPE learns faster on Pick Place and achieves substantially higher final success on Peg Insertion.
Figure 7: Left : Ablation study. Right : Hyperparameter sensitivity analysis of SLOPE.
Figure 8: Visualization of reward predictions. Top: Keyframes of task execution. Bottom: Ground truth vs. predicted rewards along the trajectory.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
SLOPE
Potential weight η
1.0
Quantile coefficient τ
0.55
TD-MPC2
Planning horizon H
3
MPPI iterations
Adroit: 8, others: 6
Appendix
Table 3: SLOPE’s hyperparameters.
ManiSkill3
Meta-World
Adroit
Robosuite
Real World
Time Horizon
100
100
100
100
Variable
Image Size
128 × 128
224 × 224
224 × 224
128 × 128
128 × 128
Observations
RGB × 2 + Prop.state
RGB + Prop.state
RGB × 2 + Prop.state
RGB × 2
RGB × 2 + Prop.state
Cameras
Hand + Front
Front
Front
Hand + Front
Hand + Side
Action Repeat
2
2
2
1
1
Action Dim
7
4
Door: 28, Hammer: 26
7
4
Appendix
Table 4: Summary of environment details for ManiSkill3 , Meta-World , Adroit , Robosuite , and Real World experiments. Prop.state stands for proprioceptive state of the robotic arm.
Domain
Task
Number of Demonstrations
ManiSkill3
Peg Insertion
100
Pick Place
100
Stack Cube
25
Poke Cube
5
Lift Peg Upright
5
Meta-World
Assembly
5
Appendix
Table 5: Number of demonstration trajectories used for each task in our experiments.
Figure 9: Real-robot setup.
Hyperparameter
Value
Environment & General
Image resolution
128×128
Proprioceptive state dim
4
Control frequency
10 Hz
Action dimension
4
Batch size
256
Appendix
Table 6: Hyperparameters for Real-Robot Experiments.
Figure 10: Visualization of reward landscape. We collect trajectory data during rollout as the basis for visualization. All methods are evaluated on the same set of states, using their respective reward models to predict rewards. The high-dimensional state representations are projected into a 2D space using Principal Component Analysis (PCA), and predicted rewards are plotted over a mesh grid to construct the reward surfaces. To avoid distortion from extreme values caused by successful states (which yield disproportionately high predicted rewards), we exclude them and visualize only states with ground-truth rewards of zero.
Figure 11: Visualization of the shaped reward landscape evolution on the 10×10 GridWorld. Darker blue regions indicate higher reward values. As training progresses, the dense reward signal propagates from the goal state (bottom-right) backwards to the initial state (top-left), progressively forming a global gradient field.
Figure 12: Success rate comparison on 20 tasks. We conduct all experiments using 5 distinct random seeds {1,2,3,4,5} .
Figure 13: Performance comparison on 8 tasks from the DMC with dense rewards. We compare SLOPE (Ours) against TD-MPC2, Dreamerv3, and SAC. All results are averaged over 5 random seeds, with shaded regions indicating 95% CIs. SLOPE consistently achieves faster convergence and higher asymptotic performance compared to the baselines, demonstrating that bootstrapped potential shaping accelerates learning even in dense reward settings.
Figure 14: Performance comparison between Dreamerv3 + SLOPE and vanilla Dreamerv3 on Meta-World sparse reward tasks. By injecting the potential-based shaping signal into the critic’s target computation during latent imagination, SLOPE significantly accelerates learning and improves final performance compared to the strong baseline.
Figure 15: Success-rate of LS-Imagine with and without SLOPE on Harvest water with bucket .
Figure 16: Learning curves on long-horizon exploration tasks. We evaluate SLOPE with DreamerV3 on Montezuma’s Revenge over ten million environment steps and with TD-MPC2 on HumanoidBench over two million interaction steps. Both HumanoidBench tasks have a 1,000-step episode horizon.
Figure 17: Performance of SLOPE without demonstrations in fully sparse reward settings.
Figure 18: Left : Performance evaluation of SLOPE in semi-sparse reward settings. Right : Comparison of different reward shaping methods.
Figure 19: Comparative analysis between SLOPE and Reward Smoothing.
Figure 20: Effect of the quantile coefficient on three sparse-reward tasks. Mild optimistic weighting ( τ=0.55 , orange) improves learning speed and average final success rates compared with the symmetric objective ( τ=0.5 , blue). The largest final-performance gain occurs on Pick Out of Hole.
Figure 21: Separate ablations of potential shaping and optimistic learning on Peg Insertion and Pick Place. Potential shaping alone substantially improves over MoDem, while adding optimistic learning further improves final performance on Peg Insertion and accelerates learning on Pick Place.
Method
Training time ↓
CPU memory ↓
Learnable parameters ↓
Average success rate ↑
TD-MPC2
7.9h
29.53GB
5.6M
0%
MoDem
9.7h
30.48GB
5.6M
55.0%
DEMO3-semi-sparse
11.8h
30.48GB
5.7M
74.0%
DEMO3-sparse
11.8h
30.48GB
5.7M
54.8%
SLOPE(Ours)
9.9h
40.37GB
5.6M
92.0%
Appendix
Table 7: Comparison of training resources for different algorithms. The best score is marked with bold . The direction of the arrow indicates whether the desired metric is large or small.
Figure 22: Visualization of successful policy rollouts in the real world. The figure displays chronological snapshots (left to right) of the agent executing the Press Button (top row), Push Cube (middle row), and Grasp Cube (bottom row) tasks. The final column, marked with a red dashed box, highlights the successful completion state for each sparse-reward task.