What Matters for Latent Reasoning with Flow Matching

Yassine Ouali; Adrian Bulat; Georgios Tzimiropoulos

cs.LGOct 5, 2026

What Matters for Latent Reasoning with Flow Matching

Authors: Yassine Ouali, Adrian Bulat, Georgios Tzimiropoulos

Organizations: Samsung AI Cambridge · Technical University of Iasi · Queen Mary University of London

Abstract

Latent reasoning lets a large language model (LLM) think in a continuous space and verbalize only the answer. We argue that an effective latent thought must meet five requirements: it should be useful, helping produce the correct answer rather than merely changing it, diverse, so that resampling yields different reasoning trajectories, explainable, so that a decoded chain of thought (CoT) reflects reasoning the answer actually follows, refinable with more inference compute, and efficient, costing less than an explicit CoT at comparable accuracy. Current methods rarely meet these requirements: they learn shortcuts from the question, distill the explicit CoT into their weights, or imitate it one token at a time. We focus on flow matching in a learned latent space, the family we argue is best placed to meet them, and identify the training choices that make it work. The result is Flow-based Latent Reasoning (FLaRe), a simple recipe covering what the latent space encodes and how to shape it, where to train the flow, how to read out the answer, and a final stage of training on the model's own verified thoughts. A probe for each requirement shows that FLaRe improves on prior latent methods in all five. It also compares favorably with them on arithmetic benchmarks, while reaching 97% of the accuracy of explicit CoT at a quarter of its latency.

Figures & tables

Figure 1 : Flow-based Latent Reasoning (FLaRe). (a) A VAE encodes a symbolic CoT

r_{\mathrm{sym}}

into a small code

\mathbf{z}

, corrupted in training for a smooth latent space, and its decoder reads it back as

r_{\mathrm{sym}}

or, given the question

q

, as a natural language CoT

r_{\mathrm{lang}}

. (b) In stage 1, the shared flow model

\theta

learns to denoise the code given the question and to answer from its own thought

\hat{\mathbf{z}}

. (c) In stage 2, on new questions with answers only,

\theta

proposes

K

thoughts per question, and those whose decoded answer is correct are re-encoded as flow targets. The model is updated with the flow loss on these targets and an answer loss

\mathcal{L}_{\mathrm{roll}}

backpropagated through the full denoising path.

Table 3

Table 5

Part of the loop	Variant	Direct	Decoded
Stage 1		55.3	61.3
Stage 2		59.1	62.6
Training	No online flow loss	59.0	61.6
Training	No stage 1 replay	59.0	62.5
Targets	No verification	57.1	60.7
	One trace	58.8	61.8

Table 6 : Stage 2 design (%). Each ablation changes one part of the loop.

	Useful	Diverse		Explainable	Refinable	Efficient
Method	follows	pass@16	vote	values in order	$0.1\to 0.5\times$ CoT compute	speedup at % CoT acc.
Explicit CoT	97.2	–	–	–	–	$1.0\times$ at 100%
Coconut	7.3	$+4.6$	$-1.5$	19.1	$+2.8$	$2.6\times$ at 61%
CODI	31.8	$+3.5$	$-0.7$	38.5	$+6.1$	$2.0\times$ at 94%
PCCoT	27.7	$+0.8$	$-1.9$	7.8	$-10.5$	$3.9\times$ at 91%
FLaRe, stage 1	39.7	$+16.6$	$+3.3$	46.0	$+0.4$ / $+18.2$	$3.9\times$ at 93%

Table 7 : The five requirements on GSM8K. Useful: twin pairs (%) whose answer follows the injected thought. Diverse: gain over greedy with 16 samples. Explainable: questions (%) whose decoded thought has all reference intermediate results in order. Refinable: gain from 0.1 to

0.5\times

explicit CoT compute. Efficient: speedup over explicit CoT at the share of its accuracy. FLaRe: decoded reading, except refinable (direct / decoded) and efficient (direct,

S=2

	Qwen2.5-0.5B-Instruct				Llama-3.2-1B-Instruct				Llama-3.2-3B-Instruct
	IID	OOD			IID	OOD			IID	OOD
Method	GSM8K	Hard	SVAMP	MArith	GSM8K	Hard	SVAMP	MArith	GSM8K	Hard	SVAMP	MArith
Explicit CoT	59.7	17.0	61.6	98.3	62.5	14.7	66.5	97.8	72.4	20.8	73.4	98.3
No-CoT	27.3	6.4	38.5	57.8	35.0	7.4	38.1	67.8	39.2	9.3	56.7	96.7
Coconut	28.2	6.8	41.9	58.3	36.1	8.3	48.4	87.2	45.3	10.9	57.5	96.7
iCoT	32.9	7.4	41.0	66.7	36.5	8.4	40.8	72.8	42.1	9.6	50.7	91.1

Table 8 : Direct answer accuracy (%) of models trained on GSM8K-Aug, on GSM8K (IID) and on GSM8K-Hard, SVAMP and MultiArith (OOD). Bold: best latent method per column and scale. Baselines come from the original papers when reported, else from our runs (Appendix A.5 ).

Method	Family	Thought lives in	Produced by	Trained with	Decodable	Budget
Explicit CoT	Horizontal	Tokens	AR decoding	CoT SFT	Written out	Model-chosen
iCoT	Vertical	Hidden states	Single pass	CoT curriculum	No	Fixed
Pause tokens	Vertical	Hidden states	Single pass	Answer loss	No	Fixed
Coconut	Horizontal	Embeddings	AR feedback	CoT curriculum	LM head	Fixed
CODI	Horizontal	Embeddings	AR feedback	CoT distillation	LM head	Fixed
PCCoT	Parallel	Embeddings	Jacobi iteration	CoT distillation	LM head	Fixed

Table 9 : Latent reasoning methods along the design axes of the paper.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

VAE
The VAE is initialized from a model trained for one epoch on the natural language CoTs of OMI-2 ( Toshniwal et al., 2025 ) with 32 slots and a single natural language route. It is then reduced to $M=8$ slots at initialization and trained with the dual objective of equation 3 with both terms weighted equally. Token substitution draws the replacement uniformly from the vocabulary with clean decoder targets, and the latent noise and dropout act on the sampled code before decoding. The statistics $(\mathbf{m},\mathbf{s})$ are the mean and standard deviation of the posterior means of 20K training rows.
Backbones $\phi$ , $\psi$	Llama-3.2-1B / 3B, fine-tuned	Token substitution $p_{\mathrm{sub}}$	0.3 on the encoder input
Initialization	OMI-2 language VAE, $32\times 512$	Latent noise	VP, $\delta=0.7$ , on half of the codes
Latent code $M\times d$	$8\times 512$	Latent dropout $p_{\mathrm{drop}}$	0.4
Input, decoder routes	$r_{\mathrm{sym}}$ , dual routes, equal weights	Optimizer	AdamW, $(0.9,0.98)$ , wd $10^{-5}$ , clip 1.0
Training data	paired GSM8K-Aug, 367K rows	Learning rate	$10^{-4}$ encoder, $2\cdot 10^{-5}$ decoder

Appendix

Table 10 : Implementation details of the final recipe: the VAE, the flow model of stages 1 and 2, the inference and the evaluation protocol.

Figure 5 : Noise draws per code. GSM8K accuracy (%) over training for one, two, four and eight draws per code, under the direct and decoded readings, for the EMA weights (solid, filled markers) and the LIVE weights (dashed, open markers).

Method	Qwen2.5-0.5B-Instruct	Llama-3.2-1B-Instruct	Llama-3.2-3B-Instruct
Explicit CoT	T	T	R (a)
No-CoT	T	T	T
iCoT	T	T	T
Coconut	T	R (b)	T
CODI	T	P	T
PCCoT	T	P on GSM8K, R (c) on the OOD sets	T

Appendix

Table 11 : Source of each baseline number of Tab. 8 . P: printed by the paper that introduced the method, CODI ( Shen et al., 2025 ) , PCCoT ( Wu et al., 2025 ) or KaVa ( Kuzina et al., 2026 ) . T: trained by us, with LaDiR reimplemented as in Appendix A.6 . R: released weights evaluated by us, the public explicit CoT checkpoint yingfanbot/gsm-cot-llama3b (a), and those of Dilgren and Wiegreffe (2026) (b) and Wu et al. (2025) (c).

Method	Tuning	Learning rate	Epochs	Other settings
Explicit CoT	LoRA, $r=128$ , $\alpha=32$	$8\cdot 10^{-4}$ / $8\cdot 10^{-4}$ / –	10	batch 128, cosine, 3% warmup, wd 0.1
No-CoT	LoRA, $r=128$ , $\alpha=32$	$2\cdot 10^{-4}$ / $8\cdot 10^{-4}$ / $2\cdot 10^{-4}$	10	as explicit CoT, answer only
iCoT	full, single precision	$10^{-5}$	20	batch 32, 8 CoT tokens removed per epoch (Qwen: 11)
Coconut	full, bf16	$5\cdot 10^{-5}$	3 + 6 + 1	$c=1$ , a CoT stage, stages 1 to 6 and a fully latent stage, batch 128
CODI	LoRA, $r=128$ , $\alpha=32$	$8\cdot 10^{-4}$ / – / $3\cdot 10^{-4}$	10 / – / 8	6 latents, distillation weight 20, batch 128
PCCoT	LoRA, $r=128$ , $\alpha=32$	$5\cdot 10^{-4}$ / – / $2\cdot 10^{-4}$	10	24 latents, $T=3$ iterations, batch 128

Appendix

Table 12 : Recipes of our baseline runs on GSM8K-Aug. Learning rates are for Qwen2.5-0.5B / Llama-3.2-1B / Llama-3.2-3B.

Corpus	Origin	Rows	Questions
GSM8K-Aug, flow	whynlp/gsm8k-aug train	385K	385K
GSM8K-Aug, paired	aligned with whynlp/gsm8k-aug-nl	367K	367K
Diverse CoT	Qwen2.5-32B-Instruct on GSM8K-Aug questions	216K	148K
Diverse CoT, paired	same, with a verified natural language twin	205K	142K
IID data	OMI-2 augmented_gsm8k , converted	166K	39K
OOD data	OMI-2 augmented_math , converted	463K	80K

Appendix

Table 13 : Corpora used in the paper. Rows are (question, CoT, answer) triples and questions are distinct normalized question texts.

VAE training data	Recon.	Direct	Decoded
GSM8K-Aug CoT	99.1	50.7	56.9
+ Diverse CoT	99.0	48.8	54.8
+ IID data	99.0	51.6	56.9
+ OOD data	99.3	50.9	56.0
+ IID and OOD data ( $5\times$ budget)	99.6	51.1	58.3

Appendix

Table 14 : Training data of the VAE, with the flow data fixed. Reconstruction (%) and GSM8K accuracy (%) under the direct and decoded readings.

IID pair, from the augmented_gsm8k share
Question	Lucy has orange trees that produce 4 oranges each. If an orange can be sold for $2 and Lucy wants to earn$ 80, how many orange trees does she need to harvest?
$r_{\mathrm{lang}}$	Each orange tree produces 4 oranges. Each orange can be sold for $2. So the amount of money Lucy earns from each tree is 4 *$ 2 = $8. If Lucy wants to earn$ 80, then she needs to harvest $80 /$ 8 = 10 trees. Thus, Lucy needs to harvest 10 orange trees.
$r_{\mathrm{sym}}$	<< 4*2=8 >> << 80/8=10 >> << 10 >>
$a$	10
OOD pair, from the augmented_math share
Question	The greatest common divisor of $a$ , $12$ , and $21$ is $3$ . What is the smallest possible value of $a$ that is greater than $3$ ?

Appendix

Table 15 : Two published pairs, one IID and one OOD, with the natural language CoT verbatim from OMI-2 and the symbolic CoT written by the converter.

Model	Follows	Keeps original	Neither	Clean follow
Explicit CoT (reference steps)	97.2	0.2	2.5	99.9
CODI	31.8	19.8	48.4	68.4
PCCoT	27.7	20.8	51.6	62.3
Coconut	7.3	29.5	63.2	19.2
FLaRe, stage 1	39.7	8.7	51.6	91.4
FLaRe, stage 2	37.3	15.0	47.7	76.1

Appendix

Table 16: Twin test on 824 pairs. Share of pairs (%) whose answer follows the injected thought (the twin’s answer), keeps the original answer, or is neither. Clean follow: the share that follows, counted only on the pairs where the model answers both the question and its twin correctly without any injection.

Figure 6 : pass@

k

on the GSM8K test set, estimated from 32 samples per question with the unbiased estimator of Chen et al. (2021) . Our models generate flow samples from noise, the baselines add noise to their thought at the calibrated scale, and explicit CoT samples its tokens at temperature 0.7.

Model	$\sigma$	Greedy	pass@1	pass@ $K$	Majority	Distinct ans.	Committed err.
CODI	0.5	55.6	54.6	59.1	54.9	1.46	51%
PCCoT	1.0	54.1	51.9	54.9	52.2	1.31	58%
Coconut	2.0	36.0	33.8	40.6	34.5	1.89	34%
FLaRe, stage 1, decoded	–	61.2	60.1	77.8	64.4	3.41	6%
FLaRe, stage 1, direct	–	55.3	–	67.1	55.6	2.68	–
FLaRe, stage 2, decoded	–	62.6	62.0	77.3	64.4	2.89	9%

Appendix

Table 17: Sampling

K=16

thoughts per question: accuracies (%), distinct answers per question, and committed errors as a share of the greedy errors.

Table 20

Figure 7 : Accuracy against the latent computation spent at inference: thoughts for Coconut and CODI, Jacobi iterations for PCCoT, loops for the depth-recurrent model, and Euler steps for our flow model, on GSM8K-Aug and GSM8K-Aug-NL. The dotted lines mark the training budget.

Explore similar work

Jun 15, 2026cs.AI

Latent Thought Flow: Efficient Latent Reasoning in Large Language Models

Large Language Models (LLMs) increasingly rely on intermediate reasoning, yet explicit Chain-of-Thought (CoT) suffers from a linguistic space bottleneck: each thought must be decoded into tokens, causing high inference overhead. Latent reasoning moves deliberation into continuous space, but existing methods mostly learn deterministic or reward-maximizing paths, lacking a principled way to allocate probability across trajectories with different correctness and costs. We propose Latent Thought Flow (LTF), which models reasoning as variable-length continuous trajectories and trains a sampler to match a reward-induced posterior over answer quality and computation cost. We instantiate this with a continuous GFlowNet using stochastic latent transitions. To handle sparse answer supervision, we introduce an Entropy-Weighted Subtrajectory Balance objective for intermediate rewards and a reference-prior regularizer to anchor exploration. Experiments under finetuning and transfer learning settings show that LTF outperforms explicit CoT and latent reasoning baselines, improving accuracy by 9.5% while reducing reasoning length by 27.2% on average compared with strong latent reasoning baselines.

Xiandong Zou, Jing Huang, Jianshu Li +1

Singapore Management University · Ant Group

Efficient Latent Reasoning Latent Thoughts Chain-of-Thought Reasoning

Ask PDF

Jun 4, 2026cs.CL

Latent Reasoning with Normalizing Flows

Large language models often improve reasoning by generating explicit chain-of-thought (CoT), demonstrating the importance of intermediate computation. However, textual CoT forces this computation through a discrete, serial, and communication-oriented token stream: each reasoning step must be verbalized before the model can proceed, even when the underlying update is semantic, uncertain, or only partially formed. Latent reasoning offers a higher-bandwidth alternative by performing intermediate computation in compact continuous states before committing to text. Yet existing latent-reasoning methods often sacrifice key advantages that make CoT effective in autoregressive language models, including native left-to-right generation, probabilistic sampling, compatibility with KV-cache decoding, and tractable likelihood estimation. We propose NF-CoT, a latent reasoning framework that preserves these advantages by modeling continuous thoughts with normalizing flows. NF-CoT instantiates a TARFlow-style normalizing flow inside the LLM backbone, defining a tractable probability model over compact continuous thoughts distilled from explicit CoT. Continuous-thought positions are generated by an NF head, while text positions are generated by the standard LM head within the same causal stream. This design provides exact likelihoods for latent thoughts, enables probabilistic left-to-right decoding with the original KV cache, and supports direct policy-gradient optimization in the latent reasoning space. On code-generation benchmarks, NF-CoT improves pass rates over explicit-CoT and prior latent-reasoning baselines while substantially reducing intermediate-reasoning cost.

Guancheng Tu, Xiangjun Fu, Suhao Yu +5

University of Pennsylvania · UC San Diego · Meta

Efficient Latent Reasoning Chain-of-Thought Reasoning Autoregressive Language Models Normalizing Flows

Ask PDF

May 8, 2026cs.CL

LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification

Chain-of-thought (CoT) reasoning improves large language models (LLMs) on difficult tasks, but it also makes inference expensive because every intermediate step must be generated as a discrete token. Latent reasoning reduces visible token generation by propagating continuous states, yet replacing explicit derivations with latent computation can hurt tasks that require symbolic checking. We propose Latent-Then-Explicit Reasoning (LaTER), a two-stage paradigm that first performs bounded exploration in a continuous latent space and then switches to explicit CoT for verification and answer generation. In a training-free instantiation, LaTER projects final-layer hidden states back to the input embedding space, preserves the latent KV cache, and uses entropy and model-native stop-token probes to decide when to switch. We find that strong reasoning models already exhibit structured latent trajectories under this interface. On Qwen3-14B, training-free LaTER reduces total token usage by 16%-32% on several benchmarks while matching or improving accuracy on most of them; for example, it improves AIME 2025 from 70.0% to 73.3% while reducing tokens from 15,730 to 10,661. We further construct Latent-Switch-69K, a supervised corpus that pairs condensed solution intuitions with shortened explicit derivations. Fine-tuning with latent rollout and halting supervision yields additional gains: trained LaTER reaches 80.0% accuracy on AIME 2025, 10.0 points above the standard CoT baseline, while using 33% fewer tokens. Our code, data, and model are available at https://github.com/TioeAre/LaTER.

Xuan Li, Yining Wang, Yuchen Liu +7

University of Science and Technology of China · Unisound AI Technology Co., Ltd · MAIS, Institute of Automation, Chinese Academy of Sciences

Efficient Latent Reasoning Chain-of-Thought Reasoning Test Time Long Visual-Token Sequences Stochastic Exploration

Ask PDF