Reinforcement learning (RL) has substantially improved the reasoning of autoregressive and discrete-diffusion language models. Language models that generate continuous token embeddings are emerging as a strong alternative, but RL post-training for them remains understudied. We propose \method{}, a post-training framework for continuous flow language models. Such a model denoises token embeddings over multiple steps and decodes them into discrete tokens only at the last step. To obtain pathwise gradients, which backpropagate through continuous actions and have much lower variance than REINFORCE, we introduce a critic that estimates the value of intermediate token embeddings, bypassing the non-differentiable decoding step. The critic must also keep track of the value function as the policy changes; to this end, we design a critic that predicts a value from every token representation and improves policy optimization. Across four reasoning tasks, \method{} reliably improves a continuous flow language model through RL post-training and outperforms baseline methods of the same type.
Figures & tables
Figure 1: Joint actor–critic training. The denoiser serves as the actor. The critic uses a frozen copy of the denoiser backbone and a value head trained jointly with the actor.
Figure 2: Where the reward lands: (a) on the token axis in an AR model; (b) on the step axis in RL for diffusion and flow models; (c) on both in CFRL -SVG-2D: rk,i for flow step k and position i , each an estimate of the whole response’s outcome.
Model
Countdown
Sudoku
K&K
Game24
GPT-2 (124M, autoregressive)
released model
0 0.0
0 0.0
0 0.0
0 0.0
+ task SFT
45.1
0 0.8
92.7
28.0
+ token GRPO/PPO
60.9 (+15.8)
0 0.8 (+0.0)
92.8 (+0.1)
46.7 (+18.7)
LLaDA-8B-Instruct (masked diffusion)
released model
0 1.4
0 0.0
0 5.4
0 1.6
Table 1: Main results. Left: accuracy (%) under the deployment sampler, with the gain over task SFT in parentheses; shaded: recommended; bold: best flow row; blue: best per task. Right: gradient estimates on Game24 base rollouts: half-split cosine (top) and accuracy after one step (bottom).
Critic
Pearson
pick@ Vϕ
max.
final
critic input and initialisation
fine-tuned copy, reads (yk,xtk−1)
0.93
0.94
58.2
57.5
without xtk−1 in the input
0.53
0.59
47.0
12.9
copy not fine-tuned
0.65
0.82
51.7
13.4
random initialisation
0.43
0.67
37.9
0 4.1
same critic, trained differently in RL
Table 2: Critic design. Left: the 2D critic on Game24 (shaded) and variants. Offline: correlation of the value with the outcome, and success of its top-ranked rollout (chance 0.55 ). Online: RL accuracy (%), max. and final. Right: the last row over training.
Figure 3: 1D versus 2D critic. Game24 accuracy (%) over training. Left: 1D- and 2D-critic runs (thin) and their means (thick). Right: critics exchanged at iteration 250 (dashed) or not (solid); colour: the critic in charge afterwards.
Choice
Variant
Δ max.
Δ final
objective
actor ascends Vϕ instead of logit(Vϕ)
−4.0
−6.1
pooling
actor ascends the minimum over positions instead of the mean
−3.1
−2.5
sampler
trained under the cheap sampler, read at deployment
−9.4
−9.4
Table 3: Effect of the objective, the pooling and the sampler on Game24: change in accuracy (%) at the best and the final checkpoint relative to the full recipe; blue: the largest loss.
Model
Countdown
K&K
cheap
deploy
cheap
deploy
base
23.8
42.8
58.6
68.2
2D-critic ckpt (loop start)
24.3
50.5
68.8
79.8
+ amortization loop
45.6
51.4
93.8
99.1
Table 4: Amortization loop. Left: accuracy (%) under the cheap and the deployment sampler before and after the loop; bold: over 1.0 point above the loop start; shaded: the full pipeline. Right: the same three models across sampler cost.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
task
states from
rollouts
stored steps
epochs (kept)
Countdown
γ4 @16
64k
1, 2, 4, 6, 8, 10, 12, 14 of 16
12 (9)
Sudoku
γ28 @32
32k
1, 6, 12, 18, 24 of 32
12 (1)
K&K
γ56 @64
32k
1, 8, 16, 24, 32, 40, 48 of 64
10 (10)
Game24
γ56 @64
16k
1, 8, 16, 32, 48 of 64
6 (6)
Appendix
Table 5: Critic pre-training per task (Algorithm 1 , Require).
SFT epoch
1
2
3
4
5
6
7
8
9
10
Countdown
26.5
30.8
38.1
38.8
41.1
43.9
42.4
44.2
43.6
42.8
Sudoku (epochs 2–20)
0 0.6
0 0.9
0 1.1
0 1.3
0 1.3
0 1.4
0 1.4
0 1.4
0 1.7
0 1.7
K&K
28.2
46.6
68.7
82.8
87.1
88.5
91.4
92.3
92.8
92.9
Game24 (init. from the Countdown model)
19.6
26.9
22.1
21.3
19.9
21.7
21.0
20.8
20.7
21.6
Game24 (init. from vanilla GPT-2)
0 1.1
0 9.8
11.4
12.4
0 9.0
0 9.5
10.3
11.2
0 9.7
0 9.7
RL iteration
0
100
150
200
300
400
500
550
600
Appendix
Table 6: AR control (GPT-2, 124M), held-out accuracy (%) with greedy decoding. Top: supervised fine-tuning by epoch (Sudoku: epochs 2, 4, …, 20, scored by the SFT script’s solved-grid metric; the verifier gives 1.2 at epoch 20). Bottom: token-level GRPO/PPO by iteration (32 prompts × 8 samples per iteration). Bold: the highest cell of each row.
steps per batch
ratio
clip
anchor
Countdown
Game24
base
42.2
45.0
CFRL -GRR
1
—
—
—
43.1
46.8
ratio only
2
yes
—
—
42.6
50.1
ratio and clip
2
yes
0.2
—
44.7
51.1
GRPO, ratio, clip and anchor
2
yes
0.2
0.04
43.2
46.0
Appendix
Table 7: GRPO variants on the flow model, held-out accuracy (%) under the deployment sampler. All rows draw the same rollouts per iteration for the same number of iterations as the CFRL -GRR row of Table 1 .
SNR, N=256
half-split cosine
N=256
N=1024
N=2048
score-function, mean-centred A
0.061
−0.03±0.03
−0.04±0.03
0.00±0.04
score-function, GRPO A (groups of 4)
0.065
0.03±0.03
0.02±0.03
−0.03±0.03
score-function, A=1 (zero-mean control)
0.063
0.02±0.02
0.04±0.02
0.02±0.03
pathwise, 2D critic
0.177
0.79±0.03
0.94±0.01
0.97±0.00
GPT-2, token-level GRPO (Game24)
—
0.09±0.05
0.35±0.04
0.34±0.07
Appendix
Table 8: The score-function ( 22 ) and pathwise ( 23 ) estimators on Game24 rollouts of the task-SFT policy at denoising step k=24 of 64 (256 rollouts for the SNR, up to 2,048 for the half-split cosine; pathwise term with the pre-trained 2D critic). SNR: norm of the mean per-sample gradient over its root-mean-square deviation; a zero-mean estimator sits at 1/N=0.0625 , the A=1 row. Half-split cosine: agreement between the mean gradients of two disjoint halves of the rollouts (mean ± sd over 10 splits) as the number of rollouts grows; a genuine signal rises with N , a zero-mean estimator stays at 0 . Last two rows: the same statistic for GPT-2’s token-level GRPO on Game24 (the panel’s line) and on Countdown. Bold: the pathwise estimator.
η/Λ
pathwise, AdamW
score-function, AdamW
random
pathwise, raw
score-function, raw
Game24 (base 43.0 , Λ=1.30 over 750 iterations, 1601 prompts)
1/256
+0.0 [-0.5, +0.5]
+0.2 [-0.2, +0.7]
+0.5
+0.5
+0.1
1/128
+0.1 [-0.5, +0.7]
+0.3 [-0.2, +0.9]
+0.3
−0.2
+0.1
1/64
+0.0 [-0.7, +0.7]
+0.2 [-0.2, +0.7]
+0.7
−0.6
+0.3
1/32
+0.9 [+0.0, +1.9]
+0.8 [+0.2, +1.5]
+0.0
−1.3
+0.4
1/16
+1.0 [-0.2, +2.1]
+0.9 [+0.2, +1.7]
+0.4
−2.1
+0.4
Appendix
Table 9: One step from the base along each mean gradient of 1,024 rollouts. Each cell is the paired change in held-out accuracy (points) against the base under the deployment sampler with the sampling noise fixed; brackets: 95% bootstrap interval over prompts. AdamW : the direction of a first AdamW step of that gradient; raw : the gradient itself; all steps have length η .
task (random)
metric
flow step k
Countdown (0.32), 16 steps
k
1
4
8
14
pick@ Vϕ
0.35
0.56
0.97
1.00
Pearson
0.14
0.33
0.91
0.98
token AUC
0.22
0.27
0.46
0.47
Sudoku (0.53), 32 steps
k
1
8
16
24
pick@ Vϕ
0.55
0.89
0.96
0.98
Appendix
Table 10: Pre-trained 2D critics on held-out rollouts of the base policy (deployment sampler; Countdown critic trained on γ4 @16 states). pick@ Vϕ over groups of 8 (random baseline in the header); Pearson correlation of the pooled value with the 0/1 reward.
Figure 4: Per-position critic readings rk,i (colour) along the deployment-sampler trajectory (rows: denoising step k ; columns: generated positions, labelled by the decoded token for Game24 and by Sudoku row) for one correct and one incorrect held-out sample per task. The correct samples are the median-valued correct rollout; the incorrect Game24 sample is the median-valued failed rollout and the incorrect Sudoku sample the failed rollout with the fewest wrong cells (dotted lines and markers).
Figure 5: 1D against 2D critic over training: the runs of Table 1 , individual runs (thin) and their mean (thick). Top: held-out accuracy (%) at the deployment sampler of the checkpoint after that many actor iterations; dashed: the SFT base. Bottom: AUC of the jointly trained head against the outcome on the newest 256 replay-buffer states, averaged over 250-iteration windows of the training logs; dotted: the floor a0=0.9 below which Algorithm 1 gives the critic extra steps. Bottom-row curves are means over the runs. On Countdown the pooled head falls behind the moving policy and its policy turns back; on Sudoku both heads keep their AUC and the 1D critic’s policy declines from the start; on Game24 and K&K both heads keep up and the 1D critic’s gain is smaller.
measurement
setting
1D
2D
Pearson correlation of the pooled value with the outcome, held-out rollouts
pathwise SNR of the actor’s parameter gradient (Table 8 statistic)
Game24 k=16 / 24 / 32
0.19 / 0.17 / 0.20
0.18 / 0.18 / 0.15
half-split cosine of the actor’s parameter gradient
Game24 k=16 / 24 / 32
0.81 / 0.78 / 0.83
0.81 / 0.79 / 0.70
cosine between the two critics’ mean actor gradients
Game24 k=16 / 24 / 32
0.90 / 0.88 / 0.83
cosine between the two critics’ per-position gradient-norm profiles
Game24, real tokens
0.94 / 0.94 / 0.94
mean pairwise cosine of per-position state gradients ∂Vϕ/∂ztk[i] (shared direction)
Game24 k=24 / Sudoku k=12
0.03 / 0.06
0.04 / 0.07
Appendix
Table 11: 1D against 2D critic on the same states. Pre-trained critics unless a row says otherwise; k is the denoising step of the state.
Figure 6: Training dynamics of Algorithm 1 (2D critic) on Countdown (left) and K&K (right): held-out accuracy at the deployment sampler per checkpoint (blue), replay-buffer success rate (grey), and the critic’s value on the training batch (red, right axis); dotted line: SFT base.
pool sampler
covered
16-step sampler
ODE-64
deployment
Countdown
SFT base
23.8
28.5
42.8
2D-critic policy (start)
24.3
34.8
50.5
round 1
deployment sampler
35.1k
37.9
42.4
45.8
round 2
deployment sampler
31.2k
40.8
42.6
46.0
round 3
critic-guided, γ4 @16
37.2k
43.6
46.3
50.0
Appendix
Table 12: The amortization loop, held-out accuracy (%) at three samplers. Each block is one task: the SFT base, the 2D-critic policy the loop starts from, the model at the end of each round of Algorithm 2 (Countdown: all four rounds; K&K: first and last of five; Sudoku: its one round), and, indented, the controls at their last round, which run the same rounds without a critic from the SFT base, from the CFRL -GRR policy, and on the gold derivations of the same pool, and, for Countdown, two further plain rounds continued from the round-2 model. Covered : pool problems (of 50k) with at least one verified sample in that round. Game24 has no pool of unlabelled problems. Sudoku’s cheap sampler is γ14 @16 and its deployment sampler γ28 @32. Bold: the last round of each loop where it is over 1.0 point above the start.
Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Recent advances in continuous diffusion and flow-based language models (LMs) have achieved performance competitive with discrete LMs. However, existing continuous frameworks still rely on decoders supervised with cross entropy (CE) because the flow trajectories are not guaranteed to terminate at valid token embeddings. Motivated by this limitation, we introduce ConvergeFlow, an embedding-space flow-based LM, which constrains the data predictor to the convex hull of token embeddings and trains it solely with the mean squared error objective induced by flow matching. Under suitable regularity conditions, we prove that the resulting flow converges to valid token embeddings despite errors in the data predictor, enabling direct token prediction without a CE-supervised decoder. We further develop three sampling mechanisms for controlling the trade-off between the generative perplexity and entropy. Experiments on OpenWebText demonstrate that ConvergeFlow achieves performance competitive with existing continuous and discrete diffusion LMs. These findings demonstrate the potential of the flow-based paradigm for language modeling. Our code is available at https://github.com/Na-Li66/ConvergeFlow.
Na Li, Yuchen Jiao, Changxiao Cai +1
Department of Statistics and Data Science, Chinese University of Hong Kong, Hong Kong · Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA
Diffusion language models are a promising alternative to autoregressive models, yet post-training methods for them largely adapt reward-maximizing objectives. We identify a central failure mode in this setting we call trajectory locking: sampled reward-driven updates over-concentrate probability mass onto a narrow set of denoising paths, reducing coverage of alternative correct solutions under repeated sampling. To address this, we propose TraFL (Trajectory Flow baLancing), a trajectory-balance objective that trains the policy toward a reward-tilted target distribution anchored to a frozen reference model. We make this practical for diffusion language models with a diffusion-compatible sequence-level surrogate and a learned prompt-dependent normalization. Across mathematical reasoning and code generation benchmarks, TraFL is the only evaluated post-training method that improves over the base model in every benchmark-length setting, with gains that persist as the sampling budget increases. The improvements transfer to held-out evaluations: TraFL stays above the base model on Minerva Math and is the strongest method on every LiveCodeBench difficulty split.