Learning When to Think Aloud in Latent Reasoning

Zhehao Zhou; Tongrui Su; Shengyu Zhu; Da Li; Zeyu Liu; Jun Zhu; Xueqi Cheng

cs.LGNov 4, 2026

Learning When to Think Aloud in Latent Reasoning

Authors: Zhehao Zhou, Tongrui Su, Shengyu Zhu, Da Li, Zeyu Liu, Jun Zhu, Xueqi Cheng

Organizations: Beijing Institute of Technology · Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · Kuaishou Technology

Abstract

Latent Chain-of-Thought (CoT) reasoning improves efficiency by shifting intermediate computation into continuous hidden states. However, fully latent methods often struggle on precision-critical tasks such as mathematical reasoning, while existing hybrid approaches retain substantial token-generation overhead. We introduce VITAL, an importance-guided hybrid reasoning framework that selectively verbalizes informative tokens as reasoning anchors while keeping most computation latent. A lightweight gating head dynamically selects between explicit token generation and direct hidden-state feedback. VITAL constructs hybrid trajectories from explicit CoT using token- importance estimation and trains the model through curriculum-based supervised training, followed by reinforcement learning with a reward that balances correctness and conciseness. Experiments show that VITAL consistently out- performs existing fully latent reasoning methods, achieving relative average accuracy gains of 9.6% on GPT-2 and 5.1% on Llama-3.2-1B at comparable reasoning lengths. Compared with hybrid baselines, VITAL achieves higher reasoning accuracy with substantially shorter reasoning trajectories on Qwen3-1.7B.

Figures & tables

Figure 1: Architecture of VITAL. A lightweight gating head dynamically selects between explicit token generation and latent computation at each reasoning step.

Figure 2: Curriculum-based supervised training for VITAL. We construct hybrid reasoning trajectories from explicit CoT by identifying high-importance tokens as the anchors.

GPT-2
Method	In-domain		Out-of-domain						Overall
	GSM8K		GSM8K-Hard		Multi-Arith		SVAMP		Average
	Acc.(%)	#Tokens	Acc.(%)	#Tokens	Acc.(%)	#Tokens	Acc.(%)	#Tokens	Acc.(%)	#Tokens
CoT-SFT	42.7	27.6	9.0	39.1	85.0	19.6	41.6	17.6	45.2	25.9
Coconut	34.1	8.0	6.9	8.0	80.7	8.0	32.0	8.0	38.4	8.0
CODI	24.6	8.0	5.0	8.0	45.0	8.0	26.9	8.0	25.4	8.0

Table 1: Main results on in-domain and out-of-domain benchmarks. Performance is measured by accuracy (Acc., %) and average reasoning length (# Tokens).

Qwen3-1.7B
Method	GSM8K-NL		MATH		MATH500		ARC-C		Average
Method	Acc.(%)	#Tokens	Acc.(%)	#Tokens	Acc.(%)	#Tokens	Acc.(%)	#Tokens	Acc.(%)	#Tokens
CoT-SFT	69.5	81.8	35.6	321.1	35.0	302.7	57.6	541.7	49.4	311.8
HRPO	64.5	110.2	27.6	763.2	26.0	765.1	53.7	464.2	43.0	525.7
Token Assorted	60.1	73.9	26.9	183.4	27.6	253.4	66.0	88.3	45.2	149.8
VITAL (Ours)	53.5	32.5	23.4	136.6	26.2	120.1	63.2	67.8	41.6	89.3

Table 2: Results on Qwen3-1.7B. Acc.(%) denotes final-answer accuracy, and #Tokens denotes average reasoning length, including both explicit tokens and latent steps.

Figure 3: Token-category distributions of full CoT and VITAL on GSM8K-NL (Qwen3-1.7B).

Figure 4: Performance degradation under noise injection. We inject Gaussian noise

\mathcal{N}(0,\sigma^{2}I_{d})

to the hidden states during reasoning and report the accuracy result of GSM8K on Llama-3.2-1B.

Method	GSM8K-Aug	GSM8K-Hard	MultiArith	SVAMP	Avg.
SFT	69.1	36.5	98.9	77.5	71.3
Coconut	60.2	25.3	97.2	65.4	62.0
Lowest	59.2	25.1	97.8	66.1	62.1
Random	62.8	28.2	97.8	67.8	64.1
VITAL	64.1	30.1	98.3	69.2	65.4

Table 3: Accuracy (%) on four benchmarks. VITAL uses highest-importance tokens as reasoning anchors.

Figure 5: Ablation on the top-

\gamma

ratio. We vary the proportion of explicit anchors and evaluate its effect.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Training Dataset	Num. Data	Avg. CoT Tokens
GSM8K-Aug	385,620	20.3

Appendix

Table 4: Training data statistics.

Benchmark	Type	Data Size
GSM8K-Aug	In-domain	1,319
GSM8K-Hard	Out-of-domain	1,319
SVAMP	Out-of-domain	1,000
MultiArith	Out-of-domain	500

Appendix

Table 5: Evaluation benchmark statistics.

Hyperparameter	Value
Anchor ratio $\gamma$	0.2
Latent placeholders $k_{\text{latent}}$	1
Gating threshold $\tau$	0.5
Loss weight $\lambda$	0.1
Curriculum stages	3
Epochs per stage	3

Appendix

Table 6: Hyperparameters of VITAL.

Method	Model	LR	Epochs
Coconut	GPT-2	$1\times 10^{-4}$	20
	Llama-3.2-1B	$8\times 10^{-4}$	20
	Llama-3.2-3B	$8\times 10^{-4}$	20
CODI	GPT-2	$3\times 10^{-3}$	40
	Llama-3.2-1B	$8\times 10^{-4}$	10
	Llama-3.2-3B	$8\times 10^{-4}$	10

Appendix

Table 7: Configurations of the baselines. All models are trained for the specified number of epochs or 50 hours, whichever is reached first.

Panel of Figure 6 — Figure 6: An example of the constructed hybrid CoT.

Method	In-domain		Out-of-domain						Overall
	GSM8K-Aug		GSM8K-Hard		MultiArith		SVAMP		Average
	Acc. (%)	# Tokens	Acc. (%)	# Tokens	Acc. (%)	# Tokens	Acc. (%)	# Tokens	Acc. (%)	# Tokens
CoT-SFT	55.4	28.5	12.7	35.7	96.7	17.6	61.0	14.5	56.5	24.0
Coconut	36.7	8.0	8.7	8.0	90.5	8.0	37.8	8.0	43.4	8.0
CODI	47.2	8.0	11.2	8.0	93.2	8.0	56.5	8.0	52.0	8.0
CoLaR	45.4	13.5	10.1	40.4	96.3	7.3	51.8	6.0	51.1	16.8

Appendix

Table 8: Experimental results on Llama-3.2-3B. Accuracy and average reasoning chain length are reported across in-domain and out-of-domain benchmarks.

Adaptive Reasoning in Language Models

See all 49 papers

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains
May 25, 2026Hui Xie, Jie Liu, Ziyue Qiao +1Efficient Language Model Reasoning LLM Inference Acceleration
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate
Jun 15, 2026Hanyu Lin, Min Cai, Jiawei Wen +1Latent Reasoning LLM Reasoning
1Shenzhen University · University of Alberta
Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought
Apr 24, 2026Keshav Ramji, Tahira Naseem, Ramón Fernandez AstudilloEfficient Language Model Reasoning Implicit Reasoning in Language Models
IBM Research AI

May 25, 2026J/K move · Enter open · S save

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains

Hui Xie, Jie Liu, Ziyue Qiao, Joaquin Vanschore

Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China

Efficient Language Model Reasoning LLM Inference Acceleration CoT Compression LLM Reasoning Latent Reasoning Latent CoT Reasoning

Open paper AskPDF arXiv

Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models (LLMs), but incurs high inference cost due to lengthy autoregressive traces. Existing latent reasoning methods offer a promis- ing alternative, yet they often treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed and thereby degrading reasoning accuracy. In this work, we propose Selective Latent Thinking (SLT), a reasoning paradigm that selectively compresses redundant reasoning spans into latent representations while preserving precision-critical spans as explicit CoT within the same reasoning trajectory. Our primary instantiation uses a lightweight decoder to anticipate a short upcoming reasoning span and applies confidence- based gating to determine the longest span that can be reliably compressed; the resulting selective compression policy is further optimized with trajectory-level reinforcement learning to balance answer correctness against reasoning cost. Ex- periments across four mathematical reasoning benchmarks show that which spans are compressed is critical: with the same compressor, compressing arbitrary spans reduces the GSM-Hard accuracy of Qwen3-4B from 54.7% to 17.3%, whereas keeping numerical tokens explicit retains 51.8%. SLT reduces explicit reasoning length by 58.4% with only a 2.8-point accuracy drop compared to explicit CoT on Llama-3.2-1B. Notably, the principle extends beyond this design: instantiated within existing latent reasoning models via a learned compression gate, it improves accuracy at equal reasoning length (+2.6 points over Latent-SFT) and at shorter length (+5.6 points over Latent-GRPO with 13% fewer tokens).

Abstract

Adaptive Reasoning in Language Models

Explore similar work

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains

Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate

Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought