Organizations: Beijing Institute of Technology, Beijing, China · Heriot-Watt University, Edinburgh, U.K. · Shenzhen Institutes of Advanced Technology, CAS, Shenzhen, China · Microsoft Research NY, USA · Tsinghua University, Beijing, China · Jilin University, Changchun, China
Model-based reinforcement learning (MBRL) is sample-efficient but struggles in sparse reward settings. A critical bottleneck arises from the lack of informative gradients in sparse settings, where standard reward models often yield flat landscapes that struggle to guide planning. To address this challenge, we propose Shaping Landscapes with Optimistic Potential Estimates (SLOPE), a novel framework that shifts reward modeling from predicting sparse scalars to constructing informative potential landscapes. SLOPE employs optimistic distributional regression to estimate high-confidence upper bounds, which amplifies rare success signals and ensures sufficient exploration gradients. Evaluations on 30+ tasks across 5 benchmarks and real-world robotic deployments, demonstrate that SLOPE consistently outperforms leading baselines in fully sparse, semi-sparse, and dense rewards. The project code is available at https://github.com/Arya87/SLOPE.
Figures & tables
Figure 1: Key challenges in sparse-reward MBRL.
Method
Reward Shaping
Demonstrations
Learning Target
Reward Supervision
Dreamer-v3 Hafner et al. (2025)
✗
✗
Ground-truth reward
Sparse
TD-MPC2 Hansen et al. (2024)
✗
✗
Ground-truth reward
Sparse
MoDem Hansen et al. (2023)
✗
✓
Ground-truth reward
Sparse
DreamSmooth Lee et al. (2024)
✓
✗
Smoothed reward
Dense + Sparse
DEMO 3 Escoriza et al. (2025)
✓
✓
Stage-wise reward
Semi-sparse
SLOPE (Ours)
✓
✓
Potential-based reward
Sparse
Table 1: Comparison of MBRL algorithmic features under sparse-reward settings. We contrast the reward modeling paradigms and supervision types of SLOPE against major baselines, highlighting how each method transforms or utilizes sparse environmental signals.
Figure 2: Framework of SLOPE. Building upon MoDem’s multi-phase accelerated learning framework, we also introduce two training enhancements: (i) initializing MPPI sampling distribution from the prior policy πPrior , and (ii) augmenting the demonstration buffer with successful trajectories. The shaped reward r is used for training both the reward model Rθ and the Q value function Qθ . The environment input consists of multimodal observations o=(x,q) , where x denotes raw RGB images captured by the robot’s camera, and q represents the robot’s proprioceptive sensory information, e.g., the gripper state. Subsequent observations are encoded by the target encoder (illustrated in grey) which is updated via an exponential moving average (EMA) of the online encoder.
Figure 3: Toy example on a 10×10 GridWorld. Left : The original environment with sparse rewards ( +1 at goal, 0 elsewhere). Right : The dense reward signal generated by PBRS using converged optimal value function V∗ . ■ : wall.
Figure 4: Visualizations of representative tasks. Top : Simulation benchmarks including ManiSkill3 ( ms ), Meta-World ( mw ), Robosuite , and Adroit . Bottom : Real-world tasks including Press Button, Push Cube, and Grasp Cube.
Figure 5: Average success rates across 20 tasks from 4 benchmarks. Curves and shaded areas represent the mean and 95% confidence intervals (CIs) over 5 independent runs. We include both sparse and semi-sparse tasks for ManiSkill3 and Meta-World , while RoboSuite and Adroit involve sparse rewards only, following their native task designs. See Appendix C.3 for individual task details.
Method
Press Button
Push Cube
Grasp Cube
Success ( ↑ )
Intervention ( ↓ )
Success ( ↑ )
Intervention ( ↓ )
Success ( ↑ )
Intervention ( ↓ )
SAC
45.0
16.8
0.0
46.5
0.0
45.3
MoDem
65.0
10.0
15.0
18.4
35.0
24.0
DEMO 3
100.0
11.1
15.0
19.1
20.0
23.2
SLOPE (Ours)
100.0
7.3
65.0
18.2
65.0
21.3
Table 2: Comparison of methods on real-world tasks at 15 k environment steps ( ≈ 1 hour of real-world training). Results are averaged over 20 evaluation trials. Success Rate and Intervention Rate are reported in percentage (%). The best results are highlighted in colorbox .
Figure 6: Comparison with intrinsic-motivation and disagreement-based exploration baselines on sparse-reward ManiSkill3 tasks. All variants share the MoDem backbone and demonstrations. Compared with RND, Q-ensemble disagreement, and Plan2Explore-style disagreement, SLOPE learns faster on Pick Place and achieves substantially higher final success on Peg Insertion.
Figure 7: Left : Ablation study. Right : Hyperparameter sensitivity analysis of SLOPE.
Figure 8: Visualization of reward predictions. Top: Keyframes of task execution. Bottom: Ground truth vs. predicted rewards along the trajectory.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
SLOPE
Potential weight η
1.0
Quantile coefficient τ
0.55
TD-MPC2
Planning horizon H
3
MPPI iterations
Adroit: 8, others: 6
Appendix
Table 3: SLOPE’s hyperparameters.
ManiSkill3
Meta-World
Adroit
Robosuite
Real World
Time Horizon
100
100
100
100
Variable
Image Size
128 × 128
224 × 224
224 × 224
128 × 128
128 × 128
Observations
RGB × 2 + Prop.state
RGB + Prop.state
RGB × 2 + Prop.state
RGB × 2
RGB × 2 + Prop.state
Cameras
Hand + Front
Front
Front
Hand + Front
Hand + Side
Action Repeat
2
2
2
1
1
Action Dim
7
4
Door: 28, Hammer: 26
7
4
Appendix
Table 4: Summary of environment details for ManiSkill3 , Meta-World , Adroit , Robosuite , and Real World experiments. Prop.state stands for proprioceptive state of the robotic arm.
Domain
Task
Number of Demonstrations
ManiSkill3
Peg Insertion
100
Pick Place
100
Stack Cube
25
Poke Cube
5
Lift Peg Upright
5
Meta-World
Assembly
5
Appendix
Table 5: Number of demonstration trajectories used for each task in our experiments.
Figure 9: Real-robot setup.
Hyperparameter
Value
Environment & General
Image resolution
128×128
Proprioceptive state dim
4
Control frequency
10 Hz
Action dimension
4
Batch size
256
Appendix
Table 6: Hyperparameters for Real-Robot Experiments.
Figure 10: Visualization of reward landscape. We collect trajectory data during rollout as the basis for visualization. All methods are evaluated on the same set of states, using their respective reward models to predict rewards. The high-dimensional state representations are projected into a 2D space using Principal Component Analysis (PCA), and predicted rewards are plotted over a mesh grid to construct the reward surfaces. To avoid distortion from extreme values caused by successful states (which yield disproportionately high predicted rewards), we exclude them and visualize only states with ground-truth rewards of zero.
Figure 11: Visualization of the shaped reward landscape evolution on the 10×10 GridWorld. Darker blue regions indicate higher reward values. As training progresses, the dense reward signal propagates from the goal state (bottom-right) backwards to the initial state (top-left), progressively forming a global gradient field.
Figure 12: Success rate comparison on 20 tasks. We conduct all experiments using 5 distinct random seeds {1,2,3,4,5} .
Figure 13: Performance comparison on 8 tasks from the DMC with dense rewards. We compare SLOPE (Ours) against TD-MPC2, Dreamerv3, and SAC. All results are averaged over 5 random seeds, with shaded regions indicating 95% CIs. SLOPE consistently achieves faster convergence and higher asymptotic performance compared to the baselines, demonstrating that bootstrapped potential shaping accelerates learning even in dense reward settings.
Figure 14: Performance comparison between Dreamerv3 + SLOPE and vanilla Dreamerv3 on Meta-World sparse reward tasks. By injecting the potential-based shaping signal into the critic’s target computation during latent imagination, SLOPE significantly accelerates learning and improves final performance compared to the strong baseline.
Figure 15: Success-rate of LS-Imagine with and without SLOPE on Harvest water with bucket .
Figure 16: Learning curves on long-horizon exploration tasks. We evaluate SLOPE with DreamerV3 on Montezuma’s Revenge over ten million environment steps and with TD-MPC2 on HumanoidBench over two million interaction steps. Both HumanoidBench tasks have a 1,000-step episode horizon.
Figure 17: Performance of SLOPE without demonstrations in fully sparse reward settings.
Figure 18: Left : Performance evaluation of SLOPE in semi-sparse reward settings. Right : Comparison of different reward shaping methods.
Figure 19: Comparative analysis between SLOPE and Reward Smoothing.
Figure 20: Effect of the quantile coefficient on three sparse-reward tasks. Mild optimistic weighting ( τ=0.55 , orange) improves learning speed and average final success rates compared with the symmetric objective ( τ=0.5 , blue). The largest final-performance gain occurs on Pick Out of Hole.
Figure 21: Separate ablations of potential shaping and optimistic learning on Peg Insertion and Pick Place. Potential shaping alone substantially improves over MoDem, while adding optimistic learning further improves final performance on Peg Insertion and accelerates learning on Pick Place.
Method
Training time ↓
CPU memory ↓
Learnable parameters ↓
Average success rate ↑
TD-MPC2
7.9h
29.53GB
5.6M
0%
MoDem
9.7h
30.48GB
5.6M
55.0%
DEMO3-semi-sparse
11.8h
30.48GB
5.7M
74.0%
DEMO3-sparse
11.8h
30.48GB
5.7M
54.8%
SLOPE(Ours)
9.9h
40.37GB
5.6M
92.0%
Appendix
Table 7: Comparison of training resources for different algorithms. The best score is marked with bold . The direction of the arrow indicates whether the desired metric is large or small.
Figure 22: Visualization of successful policy rollouts in the real world. The figure displays chronological snapshots (left to right) of the agent executing the Press Button (top row), Push Cube (middle row), and Grasp Cube (bottom row) tasks. The final column, marked with a red dashed box, highlights the successful completion state for each sparse-reward task.
Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at https://github.com/deng-ai-lab/PRM4RL.
Sparse rewards and heterogeneous task sequences remain persistent challenges in Reinforcement Learning (RL), often resulting in slow convergence, weak generalization, and inefficient exploration. We propose Uncertainty-Aware LLM-Guided Policy Shaping (ULPS), a novel framework that integrates a calibrated Large Language Model (LLM) into the RL training loop to provide structured, uncertainty-modulated behavioral guidance. ULPS employs an A*-based oracle to synthesize optimal symbolic trajectories, which are used to fine-tune a BERT-based language model. During training, this model supplies action suggestions whose influence is conditioned on epistemic uncertainty estimated via Monte Carlo (MC) dropout. An entropy-based blending mechanism adaptively balances LLM guidance and the learned policy (via Proximal Policy Optimization, PPO), allowing the agent to prioritize reliable priors while preserving adaptability. We evaluate ULPS on the MiniGridUnlockPickup benchmark and observe consistent improvements in success rate, reward efficiency, and sample complexity over unguided, uncalibrated, and standard RL baselines. ULPS achieves more than 9% improvement in execution accuracy after fine-tuning, requires fewer environment interactions, and yields higher reward AUC. Our results demonstrate that integrating symbolic A* trajectories, pretrained language priors, and uncertainty-aware control offers a principled and effective approach to multi-task reinforcement learning in sparse-reward domains, with potential extensibility to partially observable and multi-agent settings.
Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision language model (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.
Henrik Müller, Daniel Kudenko
L3S Research Center Leibniz University Hannover Hannover, 30167, Germany