BCT: Bayes-Consistent Tempering for Flow Language Models

Daisuke Oba; Hiroki Furuta; Naoaki Okazaki

cs.LGOct 20, 2026

BCT: Bayes-Consistent Tempering for Flow Language Models

Authors: Daisuke Oba, Hiroki Furuta, Naoaki Okazaki

Organizations: Institute of Science Tokyo · AIST · NII LLMC

Abstract

Temperature scaling is typically applied as a post-hoc transformation of model probabilities. In posterior-induced flow language models, this creates two mismatches. First, posterior tempering implicitly changes both the endpoint distribution and the distribution of intermediate noisy states, while the denoiser remains trained on the original states. Second, the tempered posterior weights are combined with transport dynamics derived for the untempered flow. These mismatches are reflected empirically: lowering temperature improves GSM8K performance much less reliably in flow LMs than in discrete diffusion LMs. We introduce Bayes-Consistent Tempering (BCT), which makes temperature explicit in both the distributions and dynamics of the generative flow. Its learned realization trains the denoiser on tempered intermediate states and tempers the endpoint distribution using the corresponding tempered dynamics. We also derive training-free variants for conventionally trained checkpoints. Across hyperspherical flow LMs with two denoiser architectures and a von Mises--Fisher flow LM, BCT improves GSM8K generation across broad temperature ranges and achieves the strongest tempering performance. The gains also extend to Sudoku, demonstrating benefits on tightly constrained generation. At very low temperatures, the learned realization is also markedly more robust than the training-free variants, supporting temperature as a property of the generative flow across training and inference rather than a post-hoc probability adjustment. Code and model weights are available at https://daioba.github.io/bct .

Figures & tables

Figure 1: Conceptual overview of Bayes-Consistent Tempering (BCT). Posterior tempering implicitly changes both the endpoint distribution and the conditional bridge, while standard sampling retains the original conditional velocities and denoiser training states. BCT restores consistency between the bridge and sampling dynamics, and its learned realization additionally aligns training with the tempered bridge.

Panel of Figure 2 — Figure 2: GSM8K accuracy against the sampling temperature on the two backbones (NFE $=1024$ ). Lines are the mean over three generation seeds and bands $\pm$ one sample standard deviation. Curves without a prefix use the released checkpoint; Cont. (dashed) is the matched continuation. At $T=1$ every BCT correction is the identity. Values in Table 1 .

	TFT	Cont.			$\mathbb{S}$ -FLM
$T$	BCT-L	Standard	BCT-E	BCT-P	Standard	BCT-E	BCT-P
S-arch
1	12.6 $\pm$ 1.1	12.0 $\pm$ 0.5	12.0 $\pm$ 0.5	12.0 $\pm$ 0.5	12.5 $\pm$ 0.5	12.5 $\pm$ 0.5	12.5 $\pm$ 0.5
0.8	16.3 $\pm$ 0.2	14.1 $\pm$ 0.5	15.0 $\pm$ 0.5	15.9 $\pm$ 0.2	14.7 $\pm$ 0.7	15.0 $\pm$ 1.1	16.7 $\pm$ 0.2
0.6	20.4 $\pm$ 0.4	15.3 $\pm$ 0.6	16.5 $\pm$ 0.4	20.2 $\pm$ 0.3	15.6 $\pm$ 0.2	16.9 $\pm$ 0.3	19.7 $\pm$ 0.5
0.5	21.5 $\pm$ 0.3	16.8 $\pm$ 0.2	17.4 $\pm$ 0.4	21.7 $\pm$ 0.6	16.3 $\pm$ 0.5	17.7 $\pm$ 0.3	21.2 $\pm$ 0.6

Table 1: GSM8K accuracy (%), mean

\pm

sample standard deviation over three generation seeds (1319 problems, NFE

=1024

Panel of Figure 3 — Figure 3: Pass@ $K$ on GSM8K (S-arch, NFE $=1024$ ), estimated from eight generations per problem. Bands are 95% bootstrap intervals over problems; the panels share the vertical axis.

Panel of Figure 4 — Figure 4: Sudoku-Hard exact solve rate (NFE $=180$ ): mean and $\pm$ one sample standard deviation over three generation seeds; base-checkpoint results are in Table 5 .

Panel of Figure 5 — Figure 5: GSM8K accuracy of BCT on the vMF flow LM (NFE $=1024$ ): mean and $\pm$ one sample standard deviation over five generation seeds. BCT-E falls below the plotted range for $T\leq 0.3$ ; more detailed results are in Table 7 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Held-out selection of $\lambda$ for BCT-L (634 held-out TinyGSM problems, TFT, seed 1). Thin lines: accuracy at each selection temperature; thick line: their mean, the selection score. The dotted line marks the selected $\lambda$ . $\lambda=0$ applies no Bayes correction and $\lambda=1$ the full correction. Values in Table 2 .

$\lambda$	$T{=}0.4$	$0.2$	$0.1$	Score	$T{=}0.4$	$0.2$	$0.1$	Score	Score
	S-arch				DiT				DiT (500 upd.)
0	279	285	282	44.48	271	278	287	43.95	42.27
0.25	279	292	290	45.27	278	295	316	46.74	48.74
0.5	283	304	303	46.79	281	299	295	46.00	48.32
0.75	284	305	295	46.48	282	295	257	43.85	45.79
1	286	300	289	46.00	279	279	181	38.85	43.22

Appendix

Table 2: Held-out

\lambda

sweep: correct answers out of 634 at each selection temperature and the selection score (mean accuracy over the three temperatures, %); bold: selected value. The last column is a DiT TFT checkpoint after 500 updates, not used for selection.

	Cont.				TFT
	standard tempering				BCT-L
$T$	$K=1$	$2$	$4$	$8$	$K=1$	$2$	$4$	$8$
1	12.3	19.6	28.3	37.5	12.7	20.2	29.0	37.7
0.8	14.4	22.5	32.1	42.0	16.5	25.1	34.7	44.5
0.6	15.8	24.3	34.1	44.5	20.2	29.8	40.3	51.3
0.5	16.9	25.8	36.0	46.8	21.8	31.3	41.2	51.1

Appendix

Table 3: Pass@

K

(%) of Figure 3 : unbiased estimator over

n=8

generations per problem (generation seeds

1,11,\dots,71

), S-arch, 1319 GSM8K problems. Pass@1 is the mean single-generation accuracy over the eight draws and agrees with the three-seed mean of Table 1 to within 0.6 points.

Panel of Figure 7 — Figure 7: Standard temperature scaling on the TinyGSM baselines released with $\mathbb{S}$ -FLM (NFE $=1024$ , seed 1). Gray: $\mathbb{S}$ -FLM (Section 4.2 ). Values in Table 4 .

$T$	MDLM	Duo	$\mathbb{S}$ -FLM (S-arch)	$\mathbb{S}$ -FLM (DiT)
1	15.0 (198)	18.6 (245)	13.0 (172)	11.7 (154)
0.8	22.5 (297)	23.0 (304)	15.0 (198)	13.9 (183)
0.6	28.5 (376)	28.3 (373)	15.8 (208)	16.3 (215)
0.5	29.3 (386)	29.5 (389)	16.5 (218)	17.3 (228)
0.4	30.2 (398)	32.1 (423)	17.5 (231)	17.0 (224)
0.3	32.6 (430)	34.0 (448)	18.0 (238)	16.9 (223)

Appendix

Table 4: Accuracy (%) and correct answers out of 1319 for Figure 7 .

\mathbb{S}

-FLM uses the exact velocity.

	Base checkpoint			Cont.			TFT
$T$	Standard	BCT-E	BCT-P	Standard	BCT-E	BCT-P	BCT-L
1	42.9 $\pm$ 1.5	42.9 $\pm$ 1.6	42.9 $\pm$ 1.5	41.7 $\pm$ 1.2	41.7 $\pm$ 1.2	41.7 $\pm$ 1.2	41.7 $\pm$ 1.1
0.8	42.6 $\pm$ 1.6	43.7 $\pm$ 1.4	43.6 $\pm$ 1.0	41.6 $\pm$ 1.2	42.6 $\pm$ 1.2	42.4 $\pm$ 1.0	42.7 $\pm$ 0.7
0.6	42.3 $\pm$ 1.3	44.1 $\pm$ 1.2	44.2 $\pm$ 1.1	41.4 $\pm$ 1.1	43.1 $\pm$ 1.0	42.7 $\pm$ 1.2	44.3 $\pm$ 1.2
0.5	42.2 $\pm$ 1.3	44.4 $\pm$ 1.3	44.4 $\pm$ 1.2	41.3 $\pm$ 1.0	43.3 $\pm$ 1.2	43.0 $\pm$ 0.8	45.1 $\pm$ 1.3
0.4	42.1 $\pm$ 1.2	44.8 $\pm$ 1.4	44.8 $\pm$ 1.1	41.2 $\pm$ 1.0	43.5 $\pm$ 1.2	43.2 $\pm$ 0.5	45.6 $\pm$ 1.3

Appendix

Table 5: Sudoku-Hard exact solve rate (%), mean

\pm

standard deviation over generation seeds 1, 11 and 21 (2,000 test puzzles, NFE

=180

). BCT-L uses the held-out

\lambda=1

. The seed standard deviation is dominated by a shift shared by all samplers.

$\lambda$	$T=0.4$	$T=0.2$	$T=0.1$	Mean
0	41.8	40.8	40.1	40.9
0.25	42.8	43.5	43.3	43.2
0.5	44.0	44.5	44.9	44.5
0.75	44.9	45.4	46.9	45.7
1	45.6	46.3	47.5	46.5

Appendix

Table 6: Held-out selection of

\lambda

for BCT-L on Sudoku-Hard: exact solve rate (%) of TFT on 2,000 held-out puzzles (generation seed 1, NFE

=180

). Mean: the selection score. Bold: selected. No test puzzle was used.

	Released			Cont.			TFT
$T$	Standard	BCT-E	BCT-P	Standard	BCT-E	BCT-P	BCT-L
1	23.55 $\pm$ 0.79	23.55 $\pm$ 0.79	23.55 $\pm$ 0.79	23.59 $\pm$ 0.71	23.59 $\pm$ 0.71	23.59 $\pm$ 0.71	21.55 $\pm$ 0.86
0.8	27.79 $\pm$ 0.83	29.19 $\pm$ 1.43	29.39 $\pm$ 0.48	27.99 $\pm$ 0.27	29.80 $\pm$ 0.75	29.28 $\pm$ 0.74	28.19 $\pm$ 0.97
0.6	30.34 $\pm$ 0.63	31.71 $\pm$ 0.64	31.63 $\pm$ 0.77	30.61 $\pm$ 0.77	31.54 $\pm$ 0.78	32.43 $\pm$ 0.45	32.05 $\pm$ 0.47
0.5	30.80 $\pm$ 0.80	31.40 $\pm$ 0.62	32.36 $\pm$ 1.03	31.80 $\pm$ 0.77	31.21 $\pm$ 0.47	33.28 $\pm$ 0.39	33.54 $\pm$ 0.63
0.4	31.19 $\pm$ 0.99	28.26 $\pm$ 0.52	33.39 $\pm$ 0.90	32.57 $\pm$ 0.83	26.37 $\pm$ 0.32	33.93 $\pm$ 0.58	34.33 $\pm$ 0.97

Appendix

Table 7: BCT on SFCD: GSM8K accuracy (%), mean

\pm

sample standard deviation over generation seeds 42–46 (1319 problems, NFE

=1024

). Released: the SFCD TinyGSM checkpoint with its released EMA weights. Cont. and TFT: 20,000 updates from it, evaluated with an adaptation-local EMA. BCT-L uses the held-out

\lambda=1

. At

T=1

every correction is the identity, so the three columns of a model coincide.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

Apr 6, 2026

FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation

We present FlowLM, a flow matching language model transformed from pre-trained diffusion language models via efficient fine-tuning. By re-aligning the curved sampling trajectories of diffusion models into straight-line flows, FlowLM enables high quality few-step generation that rivals or even outperforms the quality of 2,000-step diffusion sampling with very few training epochs. Remarkably, finetuned FlowLM reaches performance saturation with only half as many training epochs as training from scratch, both approaches greatly outperforming the original diffusion model, thereby validating our method. Furthermore, we validate a more effective training objective for flow matching: predicting clean data to consistently guide the sampling process towards the true data distribution. Empirical results demonstrate that our approach is highly effective for high-quality, few-step text generation.

Runzhe Zhang, Letian Chen, Wenpeng Zhang +2

Shanghai Jiao Tong University · ByteDance

Language Modeling Flow-Based Generative Modeling Diffusion Language Models Few-Step Diffusion Sampling

Ask PDF

Apr 13, 2026

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

Yuxin Chen, Chumeng Liang, Hangke Sui +4

UIUC

Self-Conditioning Diffusion Language Models Continuous Diffusion Language Models Diffusion Language Model Training Flow Matching

Ask PDF

May 11, 2026

Language Modeling with Hyperspherical Flows

Discrete Diffusion Language Models progressed rapidly as an alternative to autoregressive (AR) models, motivated by their parallel generation abilities. However, for tractability, discrete diffusion models sample from a factorized distribution, which is less expressive than AR. Recent Flow Language Models (FLMs) apply continuous flows to language, transporting noise to data with a deterministic ODE that avoids factorized sampling. FLMs operate on one-hot vectors whose dimension scales with the vocabulary size, making FLMs costly to train. Moreover, since all distinct one-hot embeddings are equidistant in

\ell_2

, adding Gaussian noise does not have a clear semantic interpretation (unlike images, where Gaussian noise progressively degrades structure). We introduce

\mathbb{S}

-FLM, a latent FLM in the hypersphere.

\mathbb{S}

-FLM generates sequences by rotating vectors in

\mathbb{S}^{d-1}

along a velocity field learned with cross-entropy, avoiding the overhead of materializing one-hot vectors. Previous FLMs match AR in Generative Perplexity (Gen.\ PPL), but samples with high likelihood are not necessarily correct in verifiable domains such as math and code.

\mathbb{S}

-FLM substantially improves continuous flow language models on large-vocabulary reasoning and closes the gap to masked diffusion under standard-temperature sampling (

T=1

), while a gap remains under optimized low-temperature (

T=0.1

) decoding.

Justin Deschenaux, Caglar Gulcehre

EPFL, Lausanne, Switzerland

Normalizing Flows Continuous Normalizing Flows Language Modeling Flow-Based Generative Modeling

Ask PDF

Abstract

Language Modeling

Explore similar work

FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Language Modeling with Hyperspherical Flows