Uncertainty-Guided Reward Labeling for Reinforcement Learning under Limited Feedback
Organizations: University of Massachusetts
Abstract
In reinforcement learning from limited feedback (RLLF), only a small fraction of an offline dataset can be labeled with rewards, and the central question is which samples should be labeled to learn a high-performing policy from the resulting partially labeled dataset. Prior work formalized this as a reward-selection problem by focusing on the selection stage while treating downstream policy learning as a black box, in a regime where queried rewards are not retained for reward-model training. We instead study the retained-label setting, where queried rewards can be stored and used to fit a reward model before policy learning. We bound the suboptimality of the learned policy in terms of two sources of error: one arising from offline RL on a fixed, offline dataset, and the other from uncertainty in the reward model. Since reward selection cannot change the offline dataset, the limited labeling budget must be used to strategically reduce reward uncertainty. Motivated by RLLF's observation that useful rewards tend to keep the agent on high-return trajectories, we introduce Successor-Guided Uncertainty REduction (SURE), a novel method that uses successor features to select rewards that are both uncertain and associated with states from which high-valued states are reachable. We formally derive SURE from a bound-induced design objective and characterize its exact one-step marginal gain. Empirically, SURE reaches near full-feedback performance with few reward labels across a variety of domains and offline-RL backbones, yielding an effective method for feedback-efficient reinforcement learning.
Figures & tables
| Environment | SURE | Exact Gain | Uniform |
| 8 / 12 / 12 | 8 / 12 / 12 | 20 / 20 / 36 | |
| 8 / 12 / 12 | 8 / 12 / 12 | 12 / 68 / 132 |
| Domain | Fraction | Percentage feedback required | |||||
| SURE | RLLF -Guided | Uniform | IDRL | Reach | Uncertainty | ||
| maze2d-medium | 95% | 0.20% | 1.74% | 2.25% | 0.32% | 1.46% | 1.96% |
| 90% | 0.12% | 1.45% | 1.79% | 0.26% | 0.50% | 1.47% | |
| 80% | 0.05% | 0.94% | 1.22% | 0.11% | 0.06% | 0.50% | |
| maze2d-large | 95% | 0.15% | 3.28% | 2.88% | 0.64% | 2.51% | 2.79% |
| 90% | 0.09% | 2.92% | 2.70% | 0.62% | 2.10% | 2.52% | |
| Backbone | Domain | SURE | Uniform |
| TD3+BC | Maze2d-Medium | 0.14% | 1.92% |
| HalfCheetah | 0.52% | 2.36% | |
| CQL | Maze2d-Medium | 0.16% | 2.00% |
| HalfCheetah | 0.55% | 2.44% |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
| Hidden sizes for , , | MLP, ReLU |
| Optimizer | Adam |
| Learning rate | (all three networks) |
| Batch size | (D4RL); (TwoRooms) |
| Discount | (HC-MR); (Maze2d-M, Maze2d-L); (TwoRooms) |
| Target-network Polyak rate |
| Hyperparameter | Value |
| Top- target-set size | (TwoRooms); (Maze2d-M); (Maze2d-L); (HC-MR) |
| Behavior-mixing weight | (constant across rounds) |
| Successor-feature ridge regularizer | |
| Successor-feature estimator | closed-form ridge (Appendix B.2 ) |
| Warm-start labels | , uniformly sampled; shared across methods and counted toward the budget |
| Acquisition batch size |
| Domain | |||
| Maze2d-Large | 0.060% | 0.064% | 0.067% |
| HalfCheetah | 0.300% | 0.330% | 0.360% |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,855 papers