Rethinking GFlowNet Training: Negative Reinforce Dynamics

Pedro Dall'Antonia; Daniel Csillag; Diego Mesquita

cs.LGOct 28, 2026

Rethinking GFlowNet Training: Negative Reinforce Dynamics

Authors: Pedro Dall'Antonia, Daniel Csillag, Diego Mesquita

Organizations: School of Applied Mathematics, Getulio Vargas Foundation

Abstract

Generative Flow Networks (GFlowNets) are models that learn to sample discrete objects proportional to a target reward. They are trained by minimizing balance errors over sampled trajectories, which inherently ties correctness to exploration: if a region is not visited, balance cannot be enforced there. Adaptive exploration is therefore crucial, yet standard training relies on a semi-gradient update that treats the sampling distribution as fixed at each optimization step. On the other hand, the full gradient reveals a second mechanism that reduces the observed errors in an anti-exploratory manner. In this paper, we establish a connection between the full gradient and min-min dynamics between the GFlowNet and a decoupled sampler, showing that the sampler reduces observed violations by avoiding high-error regions. This insight motivates a bridge between two-player games and single-policy dynamics, which underpins the design of our proposed method, Negative Reinforce Dynamics (NRD). By tying the parameters of an adversarial game together, we recover a single-policy dynamic that extends exploration while maintaining correctness. In standard benchmarks, our method outperforms existing approaches in both convergence speed and exploration efficiency, without training a separate teacher or auxiliary exploration policy.

Figures & tables

Figure 1: Two-trajectory dynamics. Left: log-ratio discrepancy over continuous time. Right: vector fields and trajectories in

(p_{1},\log Z)

from a common initialization. NRD reduces the discrepancy faster than semi-gradient TB in this example, while Naive NRD (NNRD) and the full expected-loss gradient fail to reach the balanced solution (green star).

Figure 2: Two ways to reduce the expected loss. Error correction (semi-gradient) shrinks the residual under fixed observations until it is balanced. Error avoidance (score term) leaves the residual unchanged and moves the observations onto balanced regions.

Two-player	Tied single-policy
Min-min	Full gradient
Min-max	?

Table 3

Figure 3: In the uncoupled case, the sampler

Q_{\phi}

samples trajectories and the GFlowNet corrects balance errors, while in the coupled case the GFlowNet assumes both roles.

Figure 4: Naive NRD over-allocation feedback loop. Reinforcing high-loss trajectories raises their probability regardless of the residual sign. The under-allocated trough (

\delta_{\theta}<0

) receives mass until it is balanced, whereas the over-allocated peak (

\delta_{\theta}>0

) grows and attracts even more samples. Left: before the update; right: after, with the previous residual dashed.

Figure 5: Population dynamics in exponential families. Top: phase portraits for Poisson, Gaussian location, Gaussian scale, exponential-rate, and binomial models. Bottom: Jeffreys divergence from the target along the illustrated trajectories. Time is normalized as $u=t/T\in[0,1]$ and displayed on a $\log(1+u)$ axis. These examples illustrate faster progress for NRD than semi-gradient TB and possible failure of Naive NRD or the full expected-loss gradient.

Figure 6: Distribution learning with alternative training objectives. Estimated terminal TV versus GFlowNet optimizer steps for NRD, TB, Linex(1), and Contrastive Balance. Lines and shaded bands show the mean and one standard deviation over four seeds. The GridWorld panel corresponds to the

15\times 15

HyperGrid. Lower values are better.

Figure 7: Comparison with explicit exploration mechanisms. NRD versus ACE, SA-GFN, and Adaptive Teacher on the same four distribution-learning tasks. Curves use the full optimizer-step budgets and report means with one-standard-deviation bands across four seeds. NRD has the lowest mean final estimated TV in each panel.

Figure 8: Cumulative peptide discovery. Auxiliary exploration methods above; alternative objectives below. Geometric means over four seeds.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Size / vocabulary	Batch	Iterations	Checkpoint interval
Bags18	$18/12$	32	1,500	50
Bags24	$24/12$	32	1,500	50
Knapsack128	$128/60$	32	256	10
Binary32	$32/2$	64	2,000	50
Seq24/V4	$24/4$	64	2,000	50
Seq24/V6	$24/6$	64	2,000	50

Appendix

Table 1: Structured-task configurations. Each budget is a number of training iterations for every method; checkpoint intervals use the same unit.

Environment	NRD	TB	ACE	SA-GFN	Adaptive Teacher
Bags18	$\mathbf{5.734}\pm 0.008$	$\mathbf{5.748}\pm 0.038$	$5.641\pm 0.049$	$\mathbf{5.726}\pm 0.023$	$\mathbf{5.719}\pm 0.070$
Bags24	$\mathbf{5.459}\pm 0.020$	$\mathbf{5.468}\pm 0.044$	$5.260\pm 0.075$	$\mathbf{5.395}\pm 0.073$	$\mathbf{5.455}\pm 0.077$
Knapsack128	$\mathbf{3.375}\pm 0.093$	$\mathbf{3.219}\pm 0.098$	$\mathbf{3.235}\pm 0.070$	$3.137\pm 0.064$	$3.191\pm 0.081$
Binary32	$38.897\pm 0.073$	$38.483\pm 0.028$	$\mathbf{39.010}\pm 0.497$	$38.488\pm 0.016$	$\mathbf{39.306}\pm 0.292$
Seq24/V4	$\mathbf{31.667}\pm 0.107$	$31.248\pm 0.040$	$\mathbf{32.105}\pm 0.580$	$31.221\pm 0.037$	$\mathbf{31.582}\pm 0.045$
Seq24/V6	$\mathbf{30.244}\pm 0.199$	$29.781\pm 0.073$	$\mathbf{30.252}\pm 0.265$	$29.781\pm 0.049$	$\mathbf{29.968}\pm 0.020$

Appendix

Table 2: Complete structured-discovery results at the budgets in Table 1 . Entries are the mean log reward of the top 200 unique objects, averaged over four seeds,

\pm

one sample standard deviation. Higher is better; bold marks means whose

\pm 1

SD intervals overlap that of the highest mean, using unrounded values. This describes seed variability, not statistical equivalence.

Figure 9: Distribution learning versus recorded wall time. Estimated TV for the auxiliary-policy comparison in Figure 7 , using the same source runs and seeds 39–42. Each point pairs mean elapsed time with mean TV at the matching checkpoint; shading is one sample standard deviation in TV. Elapsed time includes training and online evaluation. Curves retain their recorded endpoints, which differ across methods. Lower is better.

Estimated terminal TV
Environment	$\varepsilon=0.1$	$0.3$	$0.5$	$0.8$
Rings	$-11.14$	$2.50$	$3.88$	$-22.96$
Moons	$-34.92$	$-2.14$	$-6.13$	$-10.25$
8 Gaussians	$-13.93$	$14.03$	$31.43$	$3.71$
HyperGrid	$6.30$	$11.40$	$-1.86$	$-0.20$
FCS

Appendix

Table 3: Percentage reduction in error-curve area from using

Q_{\varepsilon}

rather than

P_{F}

in the score correction. Positive values favor

Q_{\varepsilon}

. All entries aggregate four seeds over the full prescribed budget; they are effect sizes, not significance tests. The two blocks use different error metrics.

Generative Flow Networks

Generative Flow Networks learn to sample diverse objects with probabilities proportional to their rewards.

See all 60 papers

Avoid What You Know: Divergent Trajectory Balance for GFlowNets
Feb 19, 2026Pedro Dall'Antonia, Tiago da Silva, Daniel Csillag +2Generative Flow Networks Adaptive Sampling
School of Applied Mathmatics, Getulio Vargas Foundation · MBZUAI
Proximal Policy Optimization for Amortized Discrete Sampling
Jun 14, 2026Anna Zykova-Myzina, Timofei Gritsaev, Daniil Tiapkin +1Molecular Graph Generation Generative Flow Networks
HSE University · Constructor University · CMAP, CNRS, École polytechnique, IPP
Towards Understanding the Training of GFlowNets: A Theoretical Perspective
Oct 9, 2026Rui Hu, Yu Chen, Zhuoran Li +1Generative Flow Networks Flow Matching
Institute for Interdisciplinary Information Sciences, Tsinghua University

Feb 19, 2026J/K move · Enter open · S save

Avoid What You Know: Divergent Trajectory Balance for GFlowNets

Pedro Dall'Antonia, Tiago da Silva, Daniel Csillag, Salem Lahlou+1

School of Applied Mathmatics, Getulio Vargas Foundation · MBZUAI

Generative Flow Networks Adaptive Sampling RL Exploration

Open paper AskPDF arXiv

Generative Flow Networks (GFlowNets) are a flexible family of amortized samplers trained to generate discrete and compositional objects with probability proportional to a reward function. However, learning efficiency is constrained by the model's ability to rapidly explore diverse high-probability regions during training. To mitigate this issue, recent works have focused on incentivizing the exploration of unvisited and valuable states via curiosity-driven search and self-supervised random network distillation, which tend to waste samples on already well-approximated regions of the state space. In this context, we propose Adaptive Complementary Exploration (ACE), a principled algorithm for the effective exploration of novel and high-probability regions when learning GFlowNets. To achieve this, ACE introduces an exploration GFlowNet explicitly trained to search for high-reward states in regions underexplored by the canonical GFlowNet, which learns to sample from the target distribution. Through extensive experiments, we show that ACE significantly improves upon prior work in terms of approximation accuracy to the target distribution and discovery rate of diverse high-reward states.

Abstract

Generative Flow Networks

Explore similar work

Avoid What You Know: Divergent Trajectory Balance for GFlowNets

Proximal Policy Optimization for Amortized Discrete Sampling

Towards Understanding the Training of GFlowNets: A Theoretical Perspective