Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrapping

Boyang Xu; Shengzhe Chen; Hao Yan

cs.LGSep 26, 2026

Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrapping

Authors: Boyang Xu, Shengzhe Chen, Hao Yan

Organizations: Arizona State University

Abstract

Flow critics learn return distributions by transporting Gaussian noise to Bellman endpoints via continuous velocity fields. While velocity bootstrapping stabilizes training by querying a successor teacher, existing methods face a structural dilemma: on straight paths, no residual-free same-time affine mapping can preserve Gaussian initial noise while maintaining an unbiased target. To overcome this limitation, we introduce Retimed Bellman Flows (ReBF). ReBF queries the teacher critic at a dynamically shifted earlier flow time, aligning intermediate student and teacher trajectories. By combining this retimed clock with fresh, decoupled noise generation, ReBF constructs a provably conditionally unbiased velocity target that preserves the Bellman fixed point and contracts under Wasserstein distances. Empirically, ReBF reduces $W_1$ distance to ground-truth return distributions by up to $7.7\times$ on synthetic MRPs and outperforms existing flow critics across 38 challenging OGBench and D4RL offline reinforcement learning tasks.

Figures & tables

Method	Source consistency (S)	Affinity (A)	Decoupled generation (D)
Value Flows ( Dong et al., 2026 )	✗ $R{+}\gamma\tilde{X}_{0}{\neq}X_{0}$	✓ $Z_{t}{=}R{+}\gamma X^{\prime}_{t}$	✗
PCBF ( Xu et al., 2026 )	✓ at $X_{0}$	✗ contains $X_{0}$	✗
ReBF (ours)	✓ at $X_{0}$	✓ warped affine	✓

Table 1: Comparison of flow-critic target designs across three design properties: source consistency ( S ), affinity ( A ), and decoupled generation ( D ). On the straight path, no residual-free same-time affine lift with a time-only scale holds for every endpoint pair (Corollary 3.1 ); non-affine, state-dependent, and other-path queries lie outside that statement. ReBF retimes the query and generates the successor with fresh independence. With a coupling-matched teacher and exact population regression, the target is unbiased on canonical fields.

Figure 1: Three ways to bootstrap a velocity. (a) Value Flows: same-time affine, but

Z_{0}\neq X_{0}

. (b) PCBF: source-consistent and same-time, with a residual in

X_{0}

. (c) ReBF: retimed query, independent noise.

		BCFM	PCBF	PCBF, tuned $\kappa$	floq	Value Flows	ReBF (ours)
Env	Metric	(same-time, 1, 0)	(same-time, 1, 1)	(same-time, 1, $\kappa^{\star}$ )	scalar	default	(retimed, 0, 1)
bernoulli	$W_{1}$	0.0181 $\pm$ .0023	0.0493 $\pm$ .0022	0.0180 $\pm$ .0022	0.184 $\pm$ .001	0.122 $\pm$ .004	0.0151 $\pm$ .0017
	mean	$+0.38$ $\pm$ .55	$+0.36$ $\pm$ .49	$+0.35$ $\pm$ .55	$-0.42$ $\pm$ .35	$+12.2$ $\pm$ .4	$-0.17$ $\pm$ .44
	std.	1.2 $\pm$ .2	7.2 $\pm$ .4	1.4 $\pm$ .2	13.2 $\pm$ .2	3.7 † $\pm$ .3	1.7 $\pm$ .3
solitaire	$W_{1}$	0.147 $\pm$ .006	0.677 $\pm$ .046	0.154 $\pm$ .007	1.464 $\pm$ .010	1.334 $\pm$ .010	0.0878 $\pm$ .0096
	mean	$-0.87$ $\pm$ .36	$-14.0$ $\pm$ 1.2	$-0.94$ $\pm$ .42	$+12.6$ $\pm$ .6	$-39.0$ $\pm$ .4	$-1.73$ $\pm$ .46

Table 2:

W_{1}

, mean, and std. against comparators (lower is better; mean bias near zero). Bold: best in the row; underlined: not different from the best (

p\geq 0.05

). std. is

100\sum_{s}|\mathrm{sd}_{s}-\mathrm{sd}^{\star}_{s}|/\sum_{s}\mathrm{sd}^{\star}_{s}

over states. † over-dispersed; ¶ per-coordinate floq; ‡ published Value Flows; § full maze,

2{,}880

points.

Figure 2: Why ReBF wins. (a) Effect of correction weight

\kappa

across clock and coupling variants. (b) Effect of noise coupling

\rho_{1}

at full correction (

\kappa=1

). (c) Training dynamics isolating the sequential gains of noise decoupling and clock retiming; Details in Appendix D ).

Domain	IQN	CODAC	FloQ	FQL	IQL	Value Flows	PCBF	ReBF (Ours)
cube-double-play (5 tasks)	$42\pm 8$	$61\pm 6$	$47\pm 14$	$29\pm 6$	$6\pm 2$	$69\pm 4$	$70\pm 7$	$\mathbf{76\pm 6}$
scene-play (5 tasks)	$40\pm 1$	$55\pm 1$	$\mathbf{58\pm 4}$	$56\pm 2$	$28\pm 3$	$\mathbf{59\pm 4}$	$\mathbf{58\pm 3}$	$\mathbf{60\pm 4}$
puzzle-4x4-play (5 tasks)	$27\pm 4$	$20\pm 18$	$28\pm 6$	$17\pm 5$	$7\pm 2$	$27\pm 4$	$32\pm 5$	$\mathbf{36\pm 8}$
cube-triple-play (5 tasks)	$6\pm 0$	$2\pm 1$	$8\pm 3$	$4\pm 2$	$1\pm 1$	$\mathbf{14\pm 3}$	$6\pm 3$	$12\pm 4$
D4RL adroit (8 tasks)	$66\pm 5$	$\mathbf{69\pm 1}$	$\mathbf{70\pm 3}$	$\mathbf{71\pm 4}$	$\mathbf{70}$	$65\pm 2$	$\mathbf{70\pm 2}$	$\mathbf{69\pm 1}$
visual-antmaze-teleport (5 tasks)	$4\pm 2$	–	–	$5\pm 2$	$6\pm 4$	$13\pm 4$	$\mathbf{14\pm 4}$	$13\pm 3$

Table 3: Offline RL Results. We report performance on OGBench and D4RL. Results are averaged over 8 random seeds (4 seeds for pixel-based tasks). Bold numbers denote the values within 95% of the best performing method on each domain. Per-task results are in Table 8 .

Figure 3: Comparison of learned flow fields and generated return distributions. Each plot shows velocity streamlines, sample trajectories from flow time

t=0

t=1

, and the resulting terminal return distributions (purple bars) compared against ground-truth Monte-Carlo rollouts (dashed lines) for ReBF, PCBF, Value Flows, and FLOQ.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$c=(s,a)$	Current state–action conditioning; fixed in local derivations.
$H=(R,S^{\prime},A^{\prime},\tilde{\gamma})$	Observed backup and its actual Bellman multiplier.
$X_{0},E,X_{0}^{\prime}$	Bridge source, independent auxiliary seed, and successor source.
$X_{1}^{\prime},C=X_{1}^{\prime}-X_{0}$	Successor endpoint and successor bridge displacement (the chord).
$\tau(t)$	Query clock. Same-time: $\tau(t)=t$ . ReBF: Eq. ( 6 ).
$X^{\prime}_{\tau(t)},X_{t}$	Successor interpolant at the query clock, and current Bellman interpolant.

Appendix

Table 4: Notation and distinctions used throughout the analysis.

Figure 4: Learned value and uncertainty over the whole maze , OGBench pointmaze-teleport (task $1$ , $\gamma=0.995$ ), seed $0$ , $50$ Euler steps. Columns: ReBF, PCBF, Value Flows (the published critic) and a Monte Carlo reference; top row, the mean return $V(s)$ of the law each critic generates; bottom row, its standard deviation. Each map is evaluated at $2{,}880$ points ( $8\times 8$ per free cell, spacing $0.5$ ) and linearly interpolated within free cells; walls are white. Markers are fixed maze geometry: portal entrances (circles), portal exits (triangles, one of them the sealed trap), the goal (star) and the “start” evaluation point (square). Against the reference, ReBF’s mean map has mean absolute error $13.0$ (bias $+9.3$ ) and its standard-deviation map $3.9$ , with spatial correlation $0.96$ to the reference’s and average ratio $1.05$ ; PCBF’s are $17.4$ ( $+11.9$ ), $6.5$ , $0.91$ and $0.92$ ; Value Flows’ are $60.6$ ( $+60.6$ ), $27.8$ , $0.31$ and $0.09$ , its uncertainty collapsing almost everywhere. The reference has only $30$ rollouts per point, which makes its panels grainy and puts a floor of about $4.4$ on the mean-map error and $1.7$ on the standard-deviation error; ReBF’s and PCBF’s errors are two to four times these floors. One seed: a picture, not an estimate.

Env	Split	$\kappa^{\star}$ (same-time)	$\kappa^{\star}$ (retimed)	Ratio	$p$	Verdict
bernoulli	A $\|$ B	0.1	1	1.06 $\times$	0.455	powered null
bernoulli	B $\|$ A	0.02	1	1.10 $\times$	0.292	powered null
solitaire	A $\|$ B	0	1	1.62 $\times$	0.040	retimed
solitaire	B $\|$ A	0.05	1	1.89 $\times$	0.003	retimed
discrete-mc	A $\|$ B	0.3	1	1.64 $\times$	0.004	retimed
discrete-mc	B $\|$ A	0.3	1	1.66 $\times$	0.0003	retimed

Appendix

Table 5: Tuned same-time (

\rho_{1}=1

) versus tuned retimed (

\rho_{1}=0

\kappa

selected on five seeds and scored on the five held-out seeds; both split directions shown.

W_{1}

against the exact law at

50

ODE steps,

4\times 10^{5}

draws. Every level in this table is at least

19\times

the measured evaluation floor for its environment; the gaps on solitaire and discrete-mc are significant in both split directions (

p\leq 0.04

n=5

), while on bernoulli they are

1.3

–

2.2\times

the floor at

n=15

, which is why we call it a tie (Appendix F ). Scored

n

15

on bernoulli and

5

elsewhere; ties are declared using the

t

critical value at the relevant

n

Arm	$n=1$	$n=3$	$n=5$	$n=10$	$n=20$
ReBF (retimed, $\rho_{1}{=}0$ , $\kappa{=}1$ )	$-1.64$	$-3.05$	$-1.68$	$-0.25$	$-0.42$
	0.097	0.155	0.123	0.119	0.134
PCBF (same-time, $\rho_{1}{=}1$ , $\kappa{=}1$ )	$-14.17$	$-3.94$	$-0.45$	$-0.31$	$+0.08$
	0.709	0.427	0.250	0.140	0.126
uncorrected ( $\kappa{=}0$ )	$-1.36$	$-2.48$	$-0.97$	$-0.35$	$+0.55$
	0.162	0.156	0.155	0.112	0.112

Appendix

Table 6: Mean bias (%) against the exact mean, with

W_{1}

beneath, as a function of backup depth

n

; solitaire ,

\gamma=0.9

, five seeds,

50

ODE steps. Standard errors are

0.6

–

1.4

percentage points throughout. An

n

-step target uses

\tilde{\gamma}=\gamma^{n}

and the discounted

n

-step reward.

W_{1}

here compares

4{,}000

learned draws against a

20{,}000

-draw sampled reference, whose floor is

0.043\pm 0.017

(Appendix F ); at

n=20

ReBF and PCBF differ by less than that floor’s standard deviation.

Figure 5: Every analysis in every environment (columns bernoulli , solitaire , discrete-mc and four-rooms successor features,

d=16

): (a) correction weight, (b) noise coupling, (c) training, described in Appendix D .

W_{1}

to the exact law (

4\times 10^{5}

draws) in the toy panels of rows (a) and (b), and to a

4\times 10^{5}

-rollout reference in every four-rooms panel; scoring and floors in Appendix F . Error bars are

\pm 1

s.d., except

\pm 1

s.e.m. on the solid red and green curves of the toy panels of row (a). The

\kappa=0

arm is the uncorrected critic, shown for context. The toy training curves and Value Flows (green, row a) are scored on a coarser instrument:

4\times 10^{3}

draws against a sampled

2\times 10^{4}

-draw reference, with floors of about

0.011

0.050

and

0.075

Figure 6: Four-rooms error maps , seed $0$ ( $d=16$ , $\gamma=0.95$ ; columns ReBF, PCBF, Value Flows). Row 1: mean error $\|b\|/\|\psi\|$ over all $104$ cells. Row 2: dispersion error, dropping the ill-conditioned direction $w=M^{\top}e_{15}$ , so the printed cell means are not the Table 2 entries. A star marks $(1,1)$ and circles the other evaluation states; the colour scale is set by ReBF and PCBF. Row 3: the joint law at $(1,1)$ on its two leading principal axes.

Figure 7: The $(\rho_{1},\kappa)$ surface: all $210$ cells, seeds $0$ – $4$ , $W_{1}$ to the exact law at $50$ ODE steps, one colour scale per environment. On these seeds the ReBF corner $(\rho_{1}=0,\kappa=1)$ is the lowest cell in every environment; the margin is significant only on discrete-mc ( $p=0.025$ ). At $\kappa=0$ the clock is inert, and the two panels agree to within $4.2\%$ of the cell.

Panel of Figure 8 — Figure 8: The four-rooms layout (Appendix E.4 ). Walls are dark, the four doorways light grey and the other open cells white, $104$ open cells in all. Dots mark the four evaluation states $(1,1)$ , $(4,4)$ , $(8,8)$ and $(11,11)$ (row, column; rows numbered downward).

Backup $n$	$\tilde{\gamma}=0.99^{n}$	Gaussian (any $\sigma$ )	Bernoulli law	Retimed (floor)
1	0.9900	0 0.50%	0 0.45%	$+0.09\%$
5	0.9510	0 2.48%	0 1.84%	$+0.10\%$
10	0.9044	0 4.89%	0 3.50%	$+0.10\%$
25	0.7778	11.62%	0 7.86%	$+0.13\%$
50	0.6050	20.86%	13.22%	$+0.19\%$

Appendix

Table 7: Terminal scale error of the same-time query with constant gain

\tilde{\gamma}

under an exact teacher at

\rho_{1}=0

\gamma=0.99

, as a percentage of the correct successor scale. Gaussian: Eq. ( 33 ) in closed form, identical for every

\sigma

. Bernoulli: the same constant-gain target on a

256

-atom quantile compression of the bernoulli return law, estimated by fixed-seed binned Monte Carlo (

2\times 10^{7}

draws, seed

0

100

time bins,

200

state bins,

100

Euler steps,

2\times 10^{5}

evaluation draws). Retimed: the same estimator applied to the retimed target on the Bernoulli law. Its population value is exactly zero (Theorem 5.3 and Proposition B.5 ), so this column measures the estimator’s floor, which also enters the Bernoulli column. None of these are averages over training seeds.

Figure 9: Test-time compute:

W_{1}

to the exact law (

4\times 10^{5}

draws) as the same trained critics are read out with

1

200

Euler steps (dotted line: the training resolution,

20

), mean

\pm 1

s.d. over five paired seeds; tuned PCBF uses the

\kappa^{\star}

selected on disjoint seeds. Lower is better; the

x

-axis is the readout budget, not training.

	Algorithms
Task	IQN	CODAC	FloQ	FQL	IQL	Value Flows	PCBF	ReBF (Ours)
cube-double-play-singletask-task1-v0	$70\pm 14$	$80\pm 11$	$50\pm 24$	$61\pm 9$	$27\pm 5$	$\mathbf{97\pm 1}$	$92\pm 5$	$\mathbf{97\pm 3}$
cube-double-play-singletask-task2-v0 ( $*$ )	$24\pm 9$	$63\pm 4$	$72\pm 15$	$36\pm 6$	$1\pm 1$	$76\pm 7$	$\mathbf{84\pm 8}$	$\mathbf{82\pm 9}$
cube-double-play-singletask-task3-v0	$25\pm 6$	$66\pm 9$	$57\pm 14$	$22\pm 5$	$0\pm 0$	$73\pm 4$	$80\pm 7$	$\mathbf{93\pm 2}$
cube-double-play-singletask-task4-v0	$10\pm 1$	$13\pm 2$	$8\pm 4$	$5\pm 2$	$0\pm 0$	$\mathbf{30\pm 5}$	$\mathbf{29\pm 5}$	$25\pm 6$
cube-double-play-singletask-task5-v0	$\mathbf{81\pm 8}$	$\mathbf{82\pm 4}$	$50\pm 11$	$19\pm 10$	$4\pm 3$	$69\pm 5$	$65\pm 8$	$\mathbf{83\pm 8}$

Appendix

Table 8: Full Offline RL Results. Performance on 38 OGBench ( Park et al., 2025a ) and D4RL ( Fu et al., 2020 ) tasks. The symbol

(*)

indicates the task used for hyperparameter tuning within each domain. Results for IQN ( Dabney et al., 2018a ) , CODAC ( Ma et al., 2021 ) , FloQ ( Agrawalla et al., 2026 ) , FQL ( Park et al., 2025b ) , IQL ( Kostrikov et al., 2022 ) and Value Flows ( Dong et al., 2026 ) are from Xu et al. (2026, Table 3) ; PCBF results are from the supplied PCBF comparison table. ReBF results use the best evaluation checkpoint of each seed. Bold numbers denote mean scores that are at least

95\%

of the highest reported mean on each task.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Posterior-Averaged Bellman Flow Matching for Distributional Critics
Oct 13, 2026Zeqiao Li, Yijing Wang, Haoyu Wang +2Distributional RL Flow Matching
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China
Path-Coupled Bellman Flows for Distributional Reinforcement Learning
May 7, 2026Boyang Xu, Qing Zou, Siqin Yang +1Distributional RL Flow Matching
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA
Quantile-Coupled Flow Matching for Distributional Reinforcement Learning
May 8, 2026Michael Groom, Victor-Alexandru Darvariu, Lars Kunze +2Distributional RL Conditional Flow Matching
Oxford Robotics Institute, University of Oxford, Oxford, United Kingdom · Bristol Robotics Laboratory, University of the West of England, Bristol, United Kingdom · Dyson Institute of Engineering & Technology, Malmesbury, United Kingdom

Oct 13, 2026J/K move · Enter open · S save

Posterior-Averaged Bellman Flow Matching for Distributional Critics

Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li+1

Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China

Distributional RL Flow Matching Flow-Based Generative Modeling Temporal-Difference Learning

Open paper AskPDF arXiv

Distributional critics model the full return distribution, providing richer signals for exploration and risk-sensitive control. Recent work has shown that flow-based critics can model these distributions flexibly and provide effective supervision across flow time for temporal-difference learning. However, unlike standard flow matching with fixed data endpoints, a flow critic is trained on Bellman targets recursively generated by a learned target critic under the current policy. Existing objectives therefore either use noisy single-endpoint supervision or rely on the learned flow itself for additional guidance. Our key observation is that an intermediate bridge state may be compatible with several Bellman endpoints, so its velocity should average their directions under the corresponding posterior. We propose Posterior-Averaged Bellman Flow Matching (PA-BFM), which samples an endpoint pool, infers the closed-form posterior over the latent anchor, and regresses to the posterior-mean velocity. Uniform anchor selection makes this target an exact finite-pool Rao--Blackwellization of the single-endpoint target: for every

K

, it preserves the population target, cannot increase conditional variance, and its conditional variance scales as

O(K^{-1})

. The endpoint pool supports conservative updates, while the twin return flows provide critic-disagreement and unsupported-mass diagnostics. Controlled studies show that PA-BFM reduces fixed-query target mean-squared error by

79.7\%

and target-induced gradient variance by up to

98.0\%

, while improving return-distribution fidelity. Across 50 OGBench tasks and four offline-to-online scenarios, PA-BFM achieves higher aggregate performance than Value Flows.

Abstract

Reinforcement Learning

Explore similar work

Posterior-Averaged Bellman Flow Matching for Distributional Critics

Path-Coupled Bellman Flows for Distributional Reinforcement Learning

Quantile-Coupled Flow Matching for Distributional Reinforcement Learning