Value Forcing: Aligning Return Distributions at All Noise Levels
Organizations: Princeton University · University of California, Berkeley
Abstract
Learning accurate value functions is central to reinforcement learning (RL), yet remains challenging because supervision is limited and indirect: ground-truth values are unavailable, and each transition in the data only provides an immediate reward that can be used to construct bootstrapped targets as supervision. This challenge is particularly pronounced when data is limited or reward signals are sparse, as is common in real-world applications such as robotics. We argue that the key to accurate value learning in this limited data regime is extracting richer learning signals from the available data. In this paper, we introduce Value Forcing, a new approach that uses flow matching to estimate return distributions and enables accurate value learning from limited data. Our key insight is that we can extract richer learning signals by enforcing Bellman consistency at every noise level. Empirically, Value Forcing improves the accuracy of estimating the return distribution, and prevents the distribution from collapsing to a single value, as occurs with prior methods. We instantiate Value Forcing with two practical value-learning algorithms: VF, an actor-critic method, and IVF, an implicit value-learning method. We evaluate both algorithms across long-horizon tasks in the offline RL setting and find that they outperform prior value-learning methods, with particularly large gains when data is limited. Our project website is available at https://chongyi-zheng.github.io/value-forcing.
Figures & tables
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Explicit methods | Implicit methods | |||||||||||||||||||||||||
| Environment | 00 Floq | 00000 FlowCritic | 000 Value Flows | 00 PCBF | 00 C51 | 00 IQN | 0 QC | 000 VF (Ours) | 00 DIVL | 0000 Implicit C51 | 0000 Implicit IQN | 00 Implicit QC | 000 IVF (Ours) | |||||||||||||
| cube-triple-task1 | 0 | 0 | 88 | 19 | 70 | 24 | 78 | 5 | 80 | 22 | 100 | 0 | 95 | 10 | 100 | 0 | 10 | 20 | 83 | 22 | 73 | 28 | 75 | 37 | 100 | 0 |
| cube-triple-task2 | 0 | 0 | 68 | 34 | 63 | 29 | 48 | 21 | 58 | 39 | 73 | 13 | 48 | 13 | 90 | 8 | 0 | 0 | 10 | 0 | 30 | 29 | 48 | 24 | 78 | 33 |
| cube-triple-task3 | 0 | 0 | 83 | 13 | 63 | 21 | 35 | 25 | 43 | 34 | 90 | 12 | 63 | 5 | 83 | 15 | 0 | 0 | 35 | 6 | 23 | 10 | 35 | 13 | 58 | 21 |
| cube-triple-task4 | 0 | 0 | 23 | 15 | 3 | 5 | 18 | 5 | 23 | 26 | 43 | 35 | 25 | 13 | 33 | 26 | 0 | 0 | 3 | 5 | 5 | 10 | 8 | 5 | 20 | 8 |
| cube-triple-task5 | 0 | 0 | 83 | 10 | 48 | 55 | 18 | 21 | 33 | 36 | 53 | 33 | 65 | 17 | 85 | 17 | 0 | 0 | 23 | 15 | 28 | 21 | 15 | 10 | 70 | 32 |
| Explicit methods | Implicit methods | |||||||||||||||||||||||||
| Environment (OGBench) | 00 Floq | 00000 FlowCritic | 000 Value Flows | 00 PCBF | 00 C51 | 00 IQN | 0 QC | 000 VF (Ours) | 00 DIVL | 0000 Implicit C51 | 0000 Implicit IQN | 00 Implicit QC | 000 IVF (Ours) | |||||||||||||
| cube-triple-task1 | 10 | 14 | 88 | 10 | 98 | 5 | 60 | 37 | 88 | 19 | 93 | 15 | 98 | 5 | 98 | 5 | 3 | 5 | 98 | 5 | 93 | 10 | 88 | 13 | 98 | 5 |
| cube-triple-task2 | 0 | 0 | 73 | 21 | 80 | 18 | 50 | 18 | 70 | 22 | 88 | 13 | 53 | 22 | 93 | 5 | 0 | 0 | 83 | 10 | 88 | 10 | 63 | 15 | 80 | 34 |
| cube-triple-task3 | 0 | 0 | 78 | 19 | 83 | 15 | 30 | 14 | 90 | 8 | 90 | 20 | 73 | 10 | 80 | 12 | 0 | 0 | 65 | 13 | 73 | 5 | 53 | 17 | 90 | 8 |
| cube-triple-task4 | 0 | 0 | 33 | 21 | 25 | 6 | 5 | 6 | 40 | 18 | 48 | 17 | 58 | 17 | 80 | 24 | 0 | 0 | 15 | 10 | 18 | 15 | 3 | 5 | 63 | 22 |
| cube-triple-task5 | 0 | 0 | 45 | 17 | 85 | 13 | 10 | 12 | 65 | 39 | 68 | 26 | 58 | 35 | 98 | 5 | 0 | 0 | 68 | 21 | 75 | 6 | 75 | 6 | 100 | 0 |
| Explicit methods | Implicit methods | |||||||||||||||||||||||||
| Environment | 00 Floq | 00000 FlowCritic | 000 Value Flows | 00 PCBF | 00 C51 | 00 IQN | 0 QC | 000 VF (Ours) | 00 DIVL | 0000 Implicit C51 | 0000 Implicit IQN | 00 Implicit QC | 000 IVF (Ours) | |||||||||||||
| cube-triple-task1 | 28 | 21 | 83 | 24 | 93 | 10 | 53 | 17 | 95 | 10 | 100 | 0 | 98 | 5 | 100 | 0 | 3 | 5 | 100 | 0 | 100 | 0 | 100 | 0 | 100 | 0 |
| cube-triple-task2 | 0 | 0 | 55 | 21 | 68 | 17 | 30 | 14 | 75 | 6 | 90 | 14 | 50 | 26 | 93 | 5 | 0 | 0 | 80 | 14 | 88 | 13 | 88 | 15 | 93 | 10 |
| cube-triple-task3 | 0 | 0 | 60 | 8 | 80 | 22 | 38 | 10 | 90 | 12 | 70 | 24 | 78 | 10 | 83 | 21 | 0 | 0 | 85 | 13 | 88 | 13 | 95 | 6 | 90 | 12 |
| cube-triple-task4 | 0 | 0 | 40 | 29 | 63 | 28 | 13 | 13 | 30 | 18 | 75 | 19 | 70 | 14 | 70 | 8 | 0 | 0 | 45 | 19 | 55 | 17 | 55 | 13 | 68 | 22 |
| cube-triple-task5 | 0 | 0 | 58 | 21 | 20 | 40 | 8 | 5 | 90 | 8 | 20 | 34 | 60 | 45 | 95 | 6 | 0 | 0 | 98 | 5 | 73 | 10 | 93 | 15 | 100 | 0 |
| Explicit methods | Implicit methods | |||||||||||||||||||||||||
| Environment (OGBench) | 00 Floq | 00000 FlowCritic | 000 Value Flows | 00 PCBF | 00 C51 | 00 IQN | 0 QC | 000 VF (Ours) | 00 DIVL | 0000 Implicit C51 | 0000 Implicit IQN | 00 Implicit QC | 000 IVF (Ours) | |||||||||||||
| cube-triple-task1 | 8 | 5 | 95 | 6 | 95 | 6 | 40 | 27 | 100 | 0 | 100 | 0 | 98 | 5 | 98 | 5 | 3 | 5 | 98 | 5 | 100 | 0 | 95 | 6 | 100 | 0 |
| cube-triple-task2 | 3 | 5 | 50 | 36 | 90 | 12 | 38 | 15 | 68 | 30 | 75 | 25 | 80 | 27 | 98 | 5 | 0 | 0 | 100 | 0 | 100 | 0 | 88 | 15 | 90 | 14 |
| cube-triple-task3 | 0 | 0 | 78 | 26 | 85 | 24 | 33 | 13 | 93 | 5 | 75 | 21 | 80 | 8 | 70 | 18 | 0 | 0 | 93 | 5 | 100 | 0 | 93 | 10 | 88 | 5 |
| cube-triple-task4 | 0 | 0 | 20 | 18 | 48 | 22 | 15 | 13 | 73 | 5 | 63 | 26 | 58 | 17 | 78 | 21 | 0 | 0 | 63 | 24 | 93 | 5 | 75 | 31 | 58 | 22 |
| cube-triple-task5 | 0 | 0 | 55 | 37 | 0 | 0 | 0 | 0 | 98 | 5 | 23 | 13 | 90 | 8 | 98 | 5 | 0 | 0 | 95 | 6 | 60 | 14 | 83 | 15 | 98 | 5 |
| Hyperparameter | Value |
| Optimizer | Adam ( Kingma & Ba, 2014 ) |
| Learning rate | |
| Gradient steps | 500K |
| Minibatch size | 1024 |
| Network architecture | OGBench: MLP with hidden dimensions ; OCBench: residual MLP with an input layer and two-layer residual blocks of width |
| Nonlinearity | GELU ( Hendrycks & Gimpel, 2016 ) |
| Explicit methods | |||||||
| Floq | Value Flows | PCBF | IQN | ||||
| Domain | vel-depth | ||||||
| cube-triple | 300 | 2 | 0.1 | 0.3 | 0.3 | 0.6 | 300 |
| cube-quadruple | 300 | 2 | 0.1 | 0.3 | 0.3 | 0.6 | 300 |
| puzzle-4x4 | 3000 | 4 | 0.25 | 3 | 1 | 0.4 | 300 |
| humanoidmaze-giant | 300 | 4 | 0.1 | 3 | 0.3 | 0.2 | 30 |
Value Function Estimation
Value function estimation predicts long-term returns or outcomes from states and actions to support reinforcement learning, planning, and policy evaluation.
See all 73 papers