Backpropagating Verifier Rewards Through Continuous Flow Language Models

Xu Han; Jiajing Hu; Li-Ping Liu

cs.LGOct 13, 2026

Backpropagating Verifier Rewards Through Continuous Flow Language Models

Authors: Xu Han, Jiajing Hu, Li-Ping Liu

Organizations: Tufts University

Abstract

Reinforcement learning (RL) has substantially improved the reasoning of autoregressive and discrete-diffusion language models. Language models that generate continuous token embeddings are emerging as a strong alternative, but RL post-training for them remains understudied. We propose \method{}, a post-training framework for continuous flow language models. Such a model denoises token embeddings over multiple steps and decodes them into discrete tokens only at the last step. To obtain pathwise gradients, which backpropagate through continuous actions and have much lower variance than REINFORCE, we introduce a critic that estimates the value of intermediate token embeddings, bypassing the non-differentiable decoding step. The critic must also keep track of the value function as the policy changes; to this end, we design a critic that predicts a value from every token representation and improves policy optimization. Across four reasoning tasks, \method{} reliably improves a continuous flow language model through RL post-training and outperforms baseline methods of the same type.

Figures & tables

Figure 1: Joint actor–critic training. The denoiser serves as the actor. The critic uses a frozen copy of the denoiser backbone and a value head trained jointly with the actor.

Figure 2: Where the reward lands: (a) on the token axis in an AR model; (b) on the step axis in RL for diffusion and flow models; (c) on both in CFRL -SVG-2D:

r_{k,i}

for flow step

k

and position

i

, each an estimate of the whole response’s outcome.

GPT-2 (124M, autoregressive)
Model	Countdown	Sudoku	K&K	Game24
released model	0 0.0	0 0.0	0 0.0	0 0.0
+ task SFT	45.1	0 0.8	92.7	28.0
+ token GRPO/PPO	60.9 (+15.8)	0 0.8 (+0.0)	92.8 (+0.1)	46.7 (+18.7)
LLaDA-8B-Instruct (masked diffusion)
released model	0 1.4	0 0.0	0 5.4	0 1.6

Table 1: Main results. Left: accuracy (%) under the deployment sampler, with the gain over task SFT in parentheses; shaded: recommended; bold: best flow row; blue: best per task. Right: gradient estimates on Game24 base rollouts: half-split cosine (top) and accuracy after one step (bottom).

Critic	Pearson	pick@ $V_{\phi}$	max.	final
critic input and initialisation
fine-tuned copy, reads $(y_{k},x_{t_{k-1}})$	0.93	0.94	58.2	57.5
without $x_{t_{k-1}}$ in the input	0.53	0.59	47.0	12.9
copy not fine-tuned	0.65	0.82	51.7	13.4
random initialisation	0.43	0.67	37.9	0 4.1
same critic, trained differently in RL

Table 2: Critic design. Left: the 2D critic on Game24 (shaded) and variants. Offline: correlation of the value with the outcome, and success of its top-ranked rollout (chance

0.55

). Online: RL accuracy (%), max. and final. Right: the last row over training.

Figure 3: 1D versus 2D critic. Game24 accuracy (%) over training. Left: 1D- and 2D-critic runs (thin) and their means (thick). Right: critics exchanged at iteration 250 (dashed) or not (solid); colour: the critic in charge afterwards.

Choice	Variant	$\Delta$ max.	$\Delta$ final
objective	actor ascends $V_{\phi}$ instead of $\mathrm{logit}(V_{\phi})$	$-4.0$	$-6.1$
pooling	actor ascends the minimum over positions instead of the mean	$-3.1$	$-2.5$
sampler	trained under the cheap sampler, read at deployment	$-9.4$	$-9.4$

Table 3: Effect of the objective, the pooling and the sampler on Game24: change in accuracy (%) at the best and the final checkpoint relative to the full recipe; blue: the largest loss.

Model	Countdown		K&K
	cheap	deploy	cheap	deploy
base	23.8	42.8	58.6	68.2
2D-critic ckpt (loop start)	24.3	50.5	68.8	79.8
+ amortization loop	45.6	51.4	93.8	99.1

Table 4: Amortization loop. Left: accuracy (%) under the cheap and the deployment sampler before and after the loop; bold: over 1.0 point above the loop start; shaded: the full pipeline. Right: the same three models across sampler cost.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

task	states from	rollouts	stored steps	epochs (kept)
Countdown	$\gamma 4$ @16	64k	1, 2, 4, 6, 8, 10, 12, 14 of 16	12 (9)
Sudoku	$\gamma 28$ @32	32k	1, 6, 12, 18, 24 of 32	12 (1)
K&K	$\gamma 56$ @64	32k	1, 8, 16, 24, 32, 40, 48 of 64	10 (10)
Game24	$\gamma 56$ @64	16k	1, 8, 16, 32, 48 of 64	6 (6)

Appendix

Table 5: Critic pre-training per task (Algorithm 1 , Require).

SFT epoch	1	2	3	4	5	6	7	8	9	10
Countdown	26.5	30.8	38.1	38.8	41.1	43.9	42.4	44.2	43.6	42.8
Sudoku (epochs 2–20)	0 0.6	0 0.9	0 1.1	0 1.3	0 1.3	0 1.4	0 1.4	0 1.4	0 1.7	0 1.7
K&K	28.2	46.6	68.7	82.8	87.1	88.5	91.4	92.3	92.8	92.9
Game24 (init. from the Countdown model)	19.6	26.9	22.1	21.3	19.9	21.7	21.0	20.8	20.7	21.6
Game24 (init. from vanilla GPT-2)	0 1.1	0 9.8	11.4	12.4	0 9.0	0 9.5	10.3	11.2	0 9.7	0 9.7
RL iteration	0	100	150	200	300	400	500	550	600

Appendix

Table 6: AR control (GPT-2, 124M), held-out accuracy (%) with greedy decoding. Top: supervised fine-tuning by epoch (Sudoku: epochs 2, 4, …, 20, scored by the SFT script’s solved-grid metric; the verifier gives 1.2 at epoch 20). Bottom: token-level GRPO/PPO by iteration (32 prompts

\times

8 samples per iteration). Bold: the highest cell of each row.

	steps per batch	ratio	clip	anchor	Countdown	Game24
base					42.2	45.0
CFRL -GRR	1	—	—	—	43.1	46.8
ratio only	2	yes	—	—	42.6	50.1
ratio and clip	2	yes	0.2	—	44.7	51.1
GRPO, ratio, clip and anchor	2	yes	0.2	0.04	43.2	46.0

Appendix

Table 7: GRPO variants on the flow model, held-out accuracy (%) under the deployment sampler. All rows draw the same rollouts per iteration for the same number of iterations as the CFRL -GRR row of Table 1 .

	SNR, $N{=}256$	half-split cosine
		$N{=}256$	$N{=}1024$	$N{=}2048$
score-function, mean-centred $A$	0.061	$-0.03\pm 0.03$	$-0.04\pm 0.03$	$0.00\pm 0.04$
score-function, GRPO $A$ (groups of 4)	0.065	$0.03\pm 0.03$	$0.02\pm 0.03$	$-0.03\pm 0.03$
score-function, $A{=}1$ (zero-mean control)	0.063	$0.02\pm 0.02$	$0.04\pm 0.02$	$0.02\pm 0.03$
pathwise, 2D critic	0.177	$\mathbf{0.79\pm 0.03}$	$\mathbf{0.94\pm 0.01}$	$\mathbf{0.97\pm 0.00}$
GPT-2, token-level GRPO (Game24)	—	$0.09\pm 0.05$	$0.35\pm 0.04$	$0.34\pm 0.07$

Appendix

Table 8: The score-function ( 22 ) and pathwise ( 23 ) estimators on Game24 rollouts of the task-SFT policy at denoising step

k{=}24

of 64 (256 rollouts for the SNR, up to 2,048 for the half-split cosine; pathwise term with the pre-trained 2D critic). SNR: norm of the mean per-sample gradient over its root-mean-square deviation; a zero-mean estimator sits at

1/\sqrt{N}=0.0625

, the

A{=}1

row. Half-split cosine: agreement between the mean gradients of two disjoint halves of the rollouts (mean

\pm

sd over 10 splits) as the number of rollouts grows; a genuine signal rises with

N

, a zero-mean estimator stays at

0

. Last two rows: the same statistic for GPT-2’s token-level GRPO on Game24 (the panel’s line) and on Countdown. Bold: the pathwise estimator.

Game24 (base $43.0$ , $\Lambda=1.30$ over $750$ iterations, 1601 prompts)
$\eta/\Lambda$	pathwise, AdamW	score-function, AdamW	random	pathwise, raw	score-function, raw
$1/256$	$+0.0$ [-0.5, +0.5]	$+0.2$ [-0.2, +0.7]	$+0.5$	$+0.5$	$+0.1$
$1/128$	$+0.1$ [-0.5, +0.7]	$+0.3$ [-0.2, +0.9]	$+0.3$	$-0.2$	$+0.1$
$1/64$	$+0.0$ [-0.7, +0.7]	$+0.2$ [-0.2, +0.7]	$+0.7$	$-0.6$	$+0.3$
$1/32$	$+0.9$ [+0.0, +1.9]	$+0.8$ [+0.2, +1.5]	$+0.0$	$-1.3$	$+0.4$
$1/16$	$+1.0$ [-0.2, +2.1]	$+0.9$ [+0.2, +1.7]	$+0.4$	$-2.1$	$+0.4$

Appendix

Table 9: One step from the base along each mean gradient of

1{,}024

rollouts. Each cell is the paired change in held-out accuracy (points) against the base under the deployment sampler with the sampling noise fixed; brackets:

95\%

bootstrap interval over prompts. AdamW : the direction of a first AdamW step of that gradient; raw : the gradient itself; all steps have length

\eta

task (random)	metric	flow step $k$
Countdown (0.32), 16 steps	$k$	1	4	8	14
	pick@ $V_{\phi}$	0.35	0.56	0.97	1.00
	Pearson	0.14	0.33	0.91	0.98
	token AUC	0.22	0.27	0.46	0.47
Sudoku (0.53), 32 steps	$k$	1	8	16	24
	pick@ $V_{\phi}$	0.55	0.89	0.96	0.98

Appendix

Table 10: Pre-trained 2D critics on held-out rollouts of the base policy (deployment sampler; Countdown critic trained on

\gamma 4

@16 states). pick@

V_{\phi}

over groups of 8 (random baseline in the header); Pearson correlation of the pooled value with the 0/1 reward.

Figure 4: Per-position critic readings $r_{k,i}$ (colour) along the deployment-sampler trajectory (rows: denoising step $k$ ; columns: generated positions, labelled by the decoded token for Game24 and by Sudoku row) for one correct and one incorrect held-out sample per task. The correct samples are the median-valued correct rollout; the incorrect Game24 sample is the median-valued failed rollout and the incorrect Sudoku sample the failed rollout with the fewest wrong cells (dotted lines and markers).

Figure 5: 1D against 2D critic over training: the runs of Table 1 , individual runs (thin) and their mean (thick). Top: held-out accuracy (%) at the deployment sampler of the checkpoint after that many actor iterations; dashed: the SFT base. Bottom: AUC of the jointly trained head against the outcome on the newest 256 replay-buffer states, averaged over 250-iteration windows of the training logs; dotted: the floor

a_{0}=0.9

below which Algorithm 1 gives the critic extra steps. Bottom-row curves are means over the runs. On Countdown the pooled head falls behind the moving policy and its policy turns back; on Sudoku both heads keep their AUC and the 1D critic’s policy declines from the start; on Game24 and K&K both heads keep up and the 1D critic’s gain is smaller.

measurement	setting	1D	2D
Pearson correlation of the pooled value with the outcome, held-out rollouts	pre-training, Game24 $k{=}48$ / Sudoku $k{=}18$ / Countdown $k{=}14$	0.97 / 0.94 / 0.99	0.97 / 0.94 / 0.99
pathwise SNR of the actor’s parameter gradient (Table 8 statistic)	Game24 $k{=}16$ / 24 / 32	0.19 / 0.17 / 0.20	0.18 / 0.18 / 0.15
half-split cosine of the actor’s parameter gradient	Game24 $k{=}16$ / 24 / 32	0.81 / 0.78 / 0.83	0.81 / 0.79 / 0.70
cosine between the two critics’ mean actor gradients	Game24 $k{=}16$ / 24 / 32	0.90 / 0.88 / 0.83
cosine between the two critics’ per-position gradient-norm profiles	Game24, real tokens	0.94 / 0.94 / 0.94
mean pairwise cosine of per-position state gradients $\partial V_{\phi}/\partial{\mathbf{z}}_{t_{k}}[i]$ (shared direction)	Game24 $k{=}24$ / Sudoku $k{=}12$	0.03 / 0.06	0.04 / 0.07

Appendix

Table 11: 1D against 2D critic on the same states. Pre-trained critics unless a row says otherwise;

k

is the denoising step of the state.

Figure 6: Training dynamics of Algorithm 1 (2D critic) on Countdown (left) and K&K (right): held-out accuracy at the deployment sampler per checkpoint (blue), replay-buffer success rate (grey), and the critic’s value on the training batch (red, right axis); dotted line: SFT base.

Countdown
	pool sampler	covered	16-step sampler	ODE-64	deployment
SFT base			23.8	28.5	42.8
2D-critic policy (start)			24.3	34.8	50.5
round 1	deployment sampler	35.1k	37.9	42.4	45.8
round 2	deployment sampler	31.2k	40.8	42.6	46.0
round 3	critic-guided, $\gamma 4$ @16	37.2k	43.6	46.3	50.0

Appendix

Table 12: The amortization loop, held-out accuracy (%) at three samplers. Each block is one task: the SFT base, the 2D-critic policy the loop starts from, the model at the end of each round of Algorithm 2 (Countdown: all four rounds; K&K: first and last of five; Sudoku: its one round), and, indented, the controls at their last round, which run the same rounds without a critic from the SFT base, from the CFRL -GRR policy, and on the gold derivations of the same pool, and, for Countdown, two further plain rounds continued from the round-2 model. Covered : pool problems (of 50k) with at least one verified sample in that round. Game24 has no pool of unlabelled problems. Sudoku’s cheap sampler is

\gamma 14

@16 and its deployment sampler

\gamma 28

@32. Bold: the last round of each loop where it is over 1.0 point above the start.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,089 papers

Rectified Language Flow
Oct 24, 2026Baiyu Su, Runlong Liao, Lizhang Chen +1Language Modeling Non-Autoregressive Text Generation
The University of Texas at Austin
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
Apr 13, 2026Yuxin Chen, Chumeng Liang, Hangke Sui +4Self-Conditioning Diffusion Language Models
UIUC
ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings
Aug 24, 2026Na Li, Yuchen Jiao, Changxiao Cai +1Flow-Based Generative Modeling Continuous Diffusion Language Models
Department of Statistics and Data Science, Chinese University of Hong Kong, Hong Kong · Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA

Oct 24, 2026J/K move · Enter open · S save

Rectified Language Flow

Baiyu Su, Runlong Liao, Lizhang Chen, Qiang Liu

The University of Texas at Austin

Language Modeling Non-Autoregressive Text Generation Flow-Based Generative Modeling

Open paper AskPDF arXiv

Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of autoregressive models remains challenging. We present Rectified Language Flow (ReLF), a language model that generates text by transporting noise to token embeddings using rectified flow. ReLF combines several complementary design choices: fixed token embeddings constructed with Hellinger-SVD, self-conditioning guidance trained with cross-entropy, and regularized reflow for few-step generation. We further introduce singular logits, which combine an analytically computed term that drives commitment to a valid token embedding as noise vanishes with a bounded neural residual that learns contextual token preferences. With 64 function evaluations on OpenWebText, ReLF achieves lower generative perplexity than autoregressive baselines at matched diversity across all three model sizes tested, while also outperforming prior continuous flow methods in this setting. After two reflow rounds, ReLF still outperforms autoregressive baselines with 16 function evaluations while generating 1,024-token sequences 15-19 times faster at batch size 1.

Abstract

RL for Language Model Reasoning

Explore similar work

Rectified Language Flow

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings