Learning accurate value functions is central to reinforcement learning (RL), yet remains challenging because supervision is limited and indirect: ground-truth values are unavailable, and each transition in the data only provides an immediate reward that can be used to construct bootstrapped targets as supervision. This challenge is particularly pronounced when data is limited or reward signals are sparse, as is common in real-world applications such as robotics. We argue that the key to accurate value learning in this limited data regime is extracting richer learning signals from the available data. In this paper, we introduce Value Forcing, a new approach that uses flow matching to estimate return distributions and enables accurate value learning from limited data. Our key insight is that we can extract richer learning signals by enforcing Bellman consistency at every noise level. Empirically, Value Forcing improves the accuracy of estimating the return distribution, and prevents the distribution from collapsing to a single value, as occurs with prior methods. We instantiate Value Forcing with two practical value-learning algorithms: VF, an actor-critic method, and IVF, an implicit value-learning method. We evaluate both algorithms across 26 long-horizon tasks in the offline RL setting and find that they outperform prior value-learning methods, with particularly large gains when data is limited. Our project website is available at https://chongyi-zheng.github.io/value-forcing.
Figures & tables
Figure 1: We introduce Value Forcing , a new approach to learning distributional value functions with flow matching that supervises the learned return distribution at every flow noise level through Bellman consistency. Unlike prior methods that provide supervision only at selected noise levels or use inaccurate targets at others, Value Forcing provides substantially richer learning signals and enables much more accurate estimation of the return distribution, preventing it from collapsing around a single value. This in turn leads to better policy performance, especially in the limited-data regime.
Figure 2: Value Forcing learns more accurate return distributions than prior flow-based return estimators. (Left) We collect datasets to conduct policy evaluation experiments at a single state from each of the four OGBench domains. (Center) Value Forcing captures the ground-truth return distributions more accurately, while prior methods are biased and collapse their predictions into narrow peaks. (Right) Value Forcing achieves lower mean 1 -Wasserstein distances than all three prior methods in all tasks.
Figure 3: Value Forcing improves offline RL performance across dataset sizes. Success rates are averaged over 20 OGBench tasks (Left) and 6 OCBench tasks (Right) at each dataset size. On OGBench, VF and IVF outperform their explicit and implicit baselines at 10% and 25% of the data, and remain competitive with more data. On OCBench, VF outperforms selected explicit methods on 25% of the dataset, while IVF leads among selected implicit methods on the full dataset. Hatched bars ( ) indicate distributional RL methods without flow matching. See Appendix C for complete results.
Figure 4
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Similar to the results in Sec. 6.1 , IVF learns more accurate return distributions than implicit distributional RL baselines. (Left) We conduct policy evaluation experiments on the same set of states from the four OGBench domains. (Center) IVF better captures the ground-truth return distributions, while Implicit C51, Implicit IQN, and DIVL produce shifted or overly concentrated predictions. (Right) IVF achieves lower mean 1 -Wasserstein distances than all three baselines at the evaluated state in each of the 4 domains.
Figure 7: Value Forcing learns return flows that better capture the spread of ground-truth returns. Following the policy evaluation experiments in Sec. 6.1 , we compare the return flow paths of VF ( Col 1 ), FlowCritic ( Col 2 ), Value Flows ( Col 3 ), and PCBF ( Col 4 ) from Gaussian noise ( τ=0 ) to predicted returns ( τ=1 ). VF spreads its return paths at early noise levels (around τ=0.25 ), while baselines concentrate their return paths close to the flow endpoint (around τ=0.9 ).
Figure 8: Return flow hyperparameter ablations. We conduct ablation experiments to study the sensitivity of hyperparameters related to return flows on task 2 and task 4 of cube-triple and puzzle-4x4 . For each hyperparameter, we show the success rates after aggregating over these 4 tasks.
Explicit methods
Implicit methods
Environment
00 Floq
00000 FlowCritic
000 Value Flows
00 PCBF
00 C51
00 IQN
0 QC
000 VF (Ours)
00 DIVL
0000 Implicit C51
0000 Implicit IQN
00 Implicit QC
000 IVF (Ours)
cube-triple-task1
0
± 0
88
± 19
70
± 24
78
± 5
80
± 22
100
± 0
95
± 10
100
± 0
10
± 20
83
± 22
73
± 28
75
± 37
100
± 0
cube-triple-task2
0
± 0
68
± 34
63
± 29
48
± 21
58
± 39
73
± 13
48
± 13
90
± 8
0
± 0
10
± 0
30
± 29
48
± 24
78
± 33
cube-triple-task3
0
± 0
83
± 13
63
± 21
35
± 25
43
± 34
90
± 12
63
± 5
83
± 15
0
± 0
35
± 6
23
± 10
35
± 13
58
± 21
cube-triple-task4
0
± 0
23
± 15
3
± 5
18
± 5
23
± 26
43
± 35
25
± 13
33
± 26
0
± 0
3
± 5
5
± 10
8
± 5
20
± 8
cube-triple-task5
0
± 0
83
± 10
48
± 55
18
± 21
33
± 36
53
± 33
65
± 17
85
± 17
0
± 0
23
± 15
28
± 21
15
± 10
70
± 32
Appendix
Table 2: Complete results for explicit and implicit methods with 10% of the dataset. We highlight nonzero mean performance at or above 95% of the best in each row within each method group in green .
Explicit methods
Implicit methods
Environment (OGBench)
00 Floq
00000 FlowCritic
000 Value Flows
00 PCBF
00 C51
00 IQN
0 QC
000 VF (Ours)
00 DIVL
0000 Implicit C51
0000 Implicit IQN
00 Implicit QC
000 IVF (Ours)
cube-triple-task1
10
± 14
88
± 10
98
± 5
60
± 37
88
± 19
93
± 15
98
± 5
98
± 5
3
± 5
98
± 5
93
± 10
88
± 13
98
± 5
cube-triple-task2
0
± 0
73
± 21
80
± 18
50
± 18
70
± 22
88
± 13
53
± 22
93
± 5
0
± 0
83
± 10
88
± 10
63
± 15
80
± 34
cube-triple-task3
0
± 0
78
± 19
83
± 15
30
± 14
90
± 8
90
± 20
73
± 10
80
± 12
0
± 0
65
± 13
73
± 5
53
± 17
90
± 8
cube-triple-task4
0
± 0
33
± 21
25
± 6
5
± 6
40
± 18
48
± 17
58
± 17
80
± 24
0
± 0
15
± 10
18
± 15
3
± 5
63
± 22
cube-triple-task5
0
± 0
45
± 17
85
± 13
10
± 12
65
± 39
68
± 26
58
± 35
98
± 5
0
± 0
68
± 21
75
± 6
75
± 6
100
± 0
Appendix
Table 3: Complete results for explicit and implicit methods with 25% of the dataset. We highlight nonzero mean performance at or above 95% of the best in each row within each method group in green . “ - ” marks baselines we did not run on OCBench tasks.
Explicit methods
Implicit methods
Environment
00 Floq
00000 FlowCritic
000 Value Flows
00 PCBF
00 C51
00 IQN
0 QC
000 VF (Ours)
00 DIVL
0000 Implicit C51
0000 Implicit IQN
00 Implicit QC
000 IVF (Ours)
cube-triple-task1
28
± 21
83
± 24
93
± 10
53
± 17
95
± 10
100
± 0
98
± 5
100
± 0
3
± 5
100
± 0
100
± 0
100
± 0
100
± 0
cube-triple-task2
0
± 0
55
± 21
68
± 17
30
± 14
75
± 6
90
± 14
50
± 26
93
± 5
0
± 0
80
± 14
88
± 13
88
± 15
93
± 10
cube-triple-task3
0
± 0
60
± 8
80
± 22
38
± 10
90
± 12
70
± 24
78
± 10
83
± 21
0
± 0
85
± 13
88
± 13
95
± 6
90
± 12
cube-triple-task4
0
± 0
40
± 29
63
± 28
13
± 13
30
± 18
75
± 19
70
± 14
70
± 8
0
± 0
45
± 19
55
± 17
55
± 13
68
± 22
cube-triple-task5
0
± 0
58
± 21
20
± 40
8
± 5
90
± 8
20
± 34
60
± 45
95
± 6
0
± 0
98
± 5
73
± 10
93
± 15
100
± 0
Appendix
Table 4: Complete results for explicit and implicit methods with 50% of the dataset. We highlight nonzero mean performance at or above 95% of the best in each row within each method group in green .
Explicit methods
Implicit methods
Environment (OGBench)
00 Floq
00000 FlowCritic
000 Value Flows
00 PCBF
00 C51
00 IQN
0 QC
000 VF (Ours)
00 DIVL
0000 Implicit C51
0000 Implicit IQN
00 Implicit QC
000 IVF (Ours)
cube-triple-task1
8
± 5
95
± 6
95
± 6
40
± 27
100
± 0
100
± 0
98
± 5
98
± 5
3
± 5
98
± 5
100
± 0
95
± 6
100
± 0
cube-triple-task2
3
± 5
50
± 36
90
± 12
38
± 15
68
± 30
75
± 25
80
± 27
98
± 5
0
± 0
100
± 0
100
± 0
88
± 15
90
± 14
cube-triple-task3
0
± 0
78
± 26
85
± 24
33
± 13
93
± 5
75
± 21
80
± 8
70
± 18
0
± 0
93
± 5
100
± 0
93
± 10
88
± 5
cube-triple-task4
0
± 0
20
± 18
48
± 22
15
± 13
73
± 5
63
± 26
58
± 17
78
± 21
0
± 0
63
± 24
93
± 5
75
± 31
58
± 22
cube-triple-task5
0
± 0
55
± 37
0
± 0
0
± 0
98
± 5
23
± 13
90
± 8
98
± 5
0
± 0
95
± 6
60
± 14
83
± 15
98
± 5
Appendix
Table 5: Complete results for explicit and implicit methods with 100% of the dataset. We highlight nonzero mean performance at or above 95% of the best in each row within each method group in green . “ - ” marks baselines we did not run on OCBench tasks.
Hyperparameter
Value
Optimizer
Adam ( Kingma & Ba, 2014 )
Learning rate
3×10−4
Gradient steps
500K
Minibatch size
1024
Network architecture
OGBench: MLP with hidden dimensions [1024,1024,1024,1024] ; OCBench: residual MLP with an input layer and 4 two-layer residual blocks of width 1024
Nonlinearity
GELU ( Hendrycks & Gimpel, 2016 )
Appendix
Table 6: Default hyperparameters for VF, IVF and all baselines.
Value function estimation predicts long-term returns or outcomes from states and actions to support reinforcement learning, planning, and policy evaluation.
Reliable value estimation serves as the cornerstone of reinforcement learning (RL) by evaluating long-term returns and guiding policy improvement, significantly influencing the convergence speed and final performance. Existing works improve the reliability of value function estimation via multi-critic ensembles and distributional RL, yet the former merely combines multi point estimation without capturing distributional information, whereas the latter relies on discretization or quantile regression, limiting the expressiveness of complex value distributions. Inspired by flow matching's success in generative modeling, we propose a generative paradigm for value estimation, named FlowCritic. Departing from conventional regression for deterministic value prediction, FlowCritic leverages flow matching to model value distributions and generate samples for value estimation.
Shan Zhong, Shutong Ding, He Diao +3
School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Information Science and Technology, ShanghaiTech University · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore +1
Offline-to-online reinforcement learning is promising for generalizable robotic manipulation, yet its full-stack complexity obscures reproduction and diagnosis. Within such systems, value estimation plays a central role in prioritizing heterogeneous data for policy improvement. Despite its importance, the central question remains underexplored: how value-function reliability shapes policy optimization in offline-to-online reinforcement learning. To answer this question, we propose Robo-ValueRL, a unified framework that enables reliable value estimation and systematically traces its downstream effects on policy pretraining and online improvement. Concretely, Robo-ValueRL learns a history-conditioned value estimator and evaluates its reliability through global-progress and local-preference metrics. These resulting value estimates are propagated into quality-conditioned consistency-policy pretraining and a residual adaptation module on online rollouts, providing a unified testbed for analyzing how value reliability shapes downstream policy performance. Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, our extensive experiments show that downstream performance is strongly associated with value reliability. Reliable value functions provide better action-quality estimates, allowing value-guided offline RL to scale more effectively than quality-agnostic behavior cloning, and stabilize online improvement by prioritizing high-quality rollout data. Integrating reliable value guidance through offline pretraining with online improvement, our system achieves 86% success on millimeter-level precise chip insertion and 84% on generalizable block disassembly. We hope these findings highlight the importance of value-guided data utilization for effective policy improvement from heterogeneous robotic experience.
Wenke Xia, Pei Ren, Wenbo Yu +10
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Beijing Innovation Center of Humanoid Robotics · 3Beijing Forestry University +4
We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.
Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2
The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.