Organizations: School of New Media and Communication, Tianjin University, Tianjin, China · Independent Contributor, CA, USA · School of Computer Science and Engineering, Beihang University, Beijing, China
Explicit Chain-of-Thought improves the reasoning performance of large language models but often incurs high inference cost due to verbose token-level traces. While recent approaches reduce this overhead via concise prompting or step pruning, they largely truncate what the model says rather than internalize what the model thinks. Latent reasoning offers a promising alternative by performing computation in the hidden space, yet prior methods face two critical challenges. Many existing approaches rely on rigid point-to-point alignment, forcing a latent token to approximate the final representation of a reasoning step, which can be insufficient to capture the dense, variable-length semantics of an entire reasoning segment. Furthermore, these methods often suffer from a lack of interpretability: latent states are commonly produced by unconstrained optimization or embedding mixing, yielding vectors that are difficult to decode or audit under the pretrained language head. We propose SPOT, a flexible framework that compresses explicit CoT into compact latent pause tokens without enforcing a fixed response template. At the core of SPOT is Span-level Semantic Alignment, a Sinkhorn optimal-transport objective that softly matches each pause token to the semantics of an entire reasoning segment, overcoming the rigidity of step-end alignment. To further improve interpretability, SPOT introduces a Frozen-Head Decoding Constraint that keeps latent states directly decodable as token distributions under the frozen pretrained LM head, enabling readable keyword interpretations of latent thoughts. Experiments on reasoning benchmarks demonstrate that SPOT improves accuracy by 2.3 points on average while reducing generated tokens by 37.5% and provides faithful semantic interpretations of the latent reasoning process.
Figures & tables
Figure 1. Representative coupling paradigms in hybrid/latent reasoning. Each gray block denotes a contiguous explicit reasoning span (a paragraph-level segment delimited by blank lines), and each orange block denotes a latent token.
Figure 2. Overview of the SPOT framework. Stage I trains the student on SpanDrop sequences by anchoring <pause> hidden states to the corresponding teacher span under the frozen LM head, using a Sinkhorn-based alignment loss. Stage II applies RFT by selecting correct completions and preferring shorter ones, improving stability under external <pause> insertion.
Method
GSM8K
MATH500
AIME 2024
AIME 2025
GPQA-Diamond
Average
Acc(%)
#L
Acc(%)
#L
Acc(%)
#L
Acc(%)
#L
Acc(%)
#L
Δ Acc
Δ #L
Vanilla
89.61
1,316
92.40
3,775
54.33
10,874
36.00
11,357
50.10
8,238
–
–
CCoT
89.83 +0.2%
1,226 -6.9%
92.26 -0.1%
3,133 -17.0%
52.33 -2.0%
9,884 -9.1%
35.33 -0.7%
10,842 -4.5%
50.83 +0.7%
7,869 -4.5%
-0.4%
-8.4%
DEER
91.05 +1.4%
985 -25.2%
92.60 +0.2%
2,486 -34.1%
52.67 -1.7%
9,306 -14.4%
35.67 -0.3%
10,124 -10.9%
50.65 +0.6%
5,323 -35.4%
+0.0%
-24.0%
ConciseHint
89.86 +0.3%
872 -33.7%
92.62 +0.2%
2,447 -35.2%
53.33 -1.0%
9,053 -16.8%
34.67 -1.3%
9,822 -13.5%
52.99 +2.9%
7,090 -13.9%
+0.2%
-22.6%
Step Entropy
90.46 +0.9%
1,269 -3.6%
92.80 +0.4%
2,649 -29.8%
53.33 -1.0%
8,567 -21.2%
34.67 -1.3%
9,637 -15.2%
51.98 +1.9%
7,167 -13.0%
+0.2%
-16.6%
Table 1. Main Results on DeepSeek-R1-Distill-Qwen-7B. Results are averaged over 10 random seeds. Each cell reports the absolute metric value, with subscripts showing the difference vs. Vanilla (Acc: absolute percentage points; Length: relative %). Best and second-best results are in bold and underlined , respectively.
Figure 3. Controllability under external <pause> insertion. Bars report Pass@1 accuracy ( Acc , left axis) and lines report output length ( #L , right axis) when inserting one <pause> after every N spans within the <think> segment.
Method
GSM8K
AIME2025
GPQA-Diamond
Acc (%)
#L
Acc (%)
#L
Acc (%)
#L
Vanilla
89.61
1316
36.00
11357
50.10
8238
SPOT End_KL
87.30
1204 -8.49%
21.33
10223 -9.99%
48.48
6910 -16.12%
SPOT MSE
88.93
921 -29.99%
15.67
12785 +12.58%
43.43
8586 +4.22%
SPOT Sinkhorn
92.72
630 -52.13%
39.33
9568 -15.75%
54.55
4181 -49.25%
Table 2. Ablation on the alignment objective. Each #L cell reports the absolute length with a subscript showing relative change vs. Vanilla.
G
MATH500
AIME2025
GPQA-Diamond
Acc (%)
#L
Acc (%)
#L
Acc (%)
#L
Vanilla
92.40
3775
36.00
11357
50.10
8238
1
93.80
2154 -42.95%
39.33
9568 -15.75%
54.55
4181 -49.25%
2
90.62
1675 -55.64%
32.67
9478 -16.54%
48.49
3565 -56.73%
3
89.84
1606 -57.47%
30.67
9230 -18.72%
47.65
3426 -58.42%
Table 3. Spans per <pause> . Each <pause> compresses G consecutive reasoning spans.
λ
GSM8K
AIME2025
GPQA-Diamond
Acc (%)
#L
Acc (%)
#L
Acc (%)
#L
Vanilla
89.61
1316
36.00
11357
50.10
8238
0.0
86.66
1235 -6.12%
24.00
11606 +2.20%
34.34
8192 -0.55%
0.2
88.57
1057 -19.67%
28.00
10149 -10.63%
47.47
6861 -16.71%
0.4
89.23
1082 -17.77%
29.67
9796 -13.74%
47.98
6327 -23.20%
0.6
90.00
969 -26.35%
31.33
9621 -15.28%
48.99
5870 -28.74%
Table 4. Sensitivity to the alignment weight λ .
Figure 4. Training-time alignment diagnostics on SpanDrop examples. Left axis: span-level Sinkhorn OT alignment loss (lower is better). Right axis: frozen-head top- K coverage between tokens decoded from the <pause> hidden state and the token set of the paired teacher span (Eq. 18 , K=20 ; higher is better). The dashed line denotes the vanilla backbone with externally inserted <pause> tokens evaluated under the same computation.
Method
Local ↑
Util. ↑
Joint@4 ↑ (%)
Vanilla
3.59
1.71
17.2
SPOT (End_KL)
3.65
3.20
29.7
SPOT (MSE)
3.98
2.83
30.6
SPOT (Sinkhorn)
4.85
4.34
83.6
Table 5. LLM-as-a-Judge evaluation of <pause> boundaries. We report mean scores over all evaluated boundaries. Joint@4 is the percentage of boundaries where both Local and Util. are at least 4 (out of 5).
Figure 5. A successful <pause> compression example.
Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models (LLMs), but incurs high inference cost due to lengthy autoregressive traces. Existing latent reasoning methods offer a promis- ing alternative, yet they often treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed and thereby degrading reasoning accuracy. In this work, we propose Selective Latent Thinking (SLT), a reasoning paradigm that selectively compresses redundant reasoning spans into latent representations while preserving precision-critical spans as explicit CoT within the same reasoning trajectory. Our primary instantiation uses a lightweight decoder to anticipate a short upcoming reasoning span and applies confidence- based gating to determine the longest span that can be reliably compressed; the resulting selective compression policy is further optimized with trajectory-level reinforcement learning to balance answer correctness against reasoning cost. Ex- periments across four mathematical reasoning benchmarks show that which spans are compressed is critical: with the same compressor, compressing arbitrary spans reduces the GSM-Hard accuracy of Qwen3-4B from 54.7% to 17.3%, whereas keeping numerical tokens explicit retains 51.8%. SLT reduces explicit reasoning length by 58.4% with only a 2.8-point accuracy drop compared to explicit CoT on Llama-3.2-1B. Notably, the principle extends beyond this design: instantiated within existing latent reasoning models via a learned compression gate, it improves accuracy at equal reasoning length (+2.6 points over Latent-SFT) and at shorter length (+5.6 points over Latent-GRPO with 13% fewer tokens).
Hui Xie, Jie Liu, Ziyue Qiao +1
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning through explicit Chain-of-Thought (CoT), but the associated computational overhead and token verbosity remain significant barriers to efficiency. While implicit reasoning in continuous latent spaces offers a promising alternative, existing methods like Coconut and CODI often rely on fixed-length latent sequences and suffer from representational collapse. In this paper, we propose Sep-Latent, an adaptive implicit reasoning framework that utilizes separator tokens as structural semantic anchors. By introducing an Attention-Decay mechanism, we force the model to aggregate and compress reasoning logic into these anchors during training. We then align explicit and implicit pathways through multi-level distillation across all layers to ensure logical consistency. Crucially, Sep-Latent enables dynamic reasoning depth by adaptively terminating the latent thinking phase based on query complexity. Experimental results on the GSM8k-Aug benchmark show that Sep-Latent achieves an accuracy of 56.8% with an average of only 5.4 tokens, significantly outperforming prior fixed-length implicit models in efficiency while narrowing the performance gap with explicit CoT. Further analysis via PCA and symbolic variant testing confirms that our method ensures semantic grounding and exhibits superior robustness to perturbations.
Yinlong Xu, Jianfeng Huang, Zhengdong Zhou +5
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Alibaba Group, Hangzhou, China
Large Language Models (LLMs) increasingly rely on intermediate reasoning, yet explicit Chain-of-Thought (CoT) suffers from a linguistic space bottleneck: each thought must be decoded into tokens, causing high inference overhead. Latent reasoning moves deliberation into continuous space, but existing methods mostly learn deterministic or reward-maximizing paths, lacking a principled way to allocate probability across trajectories with different correctness and costs. We propose Latent Thought Flow (LTF), which models reasoning as variable-length continuous trajectories and trains a sampler to match a reward-induced posterior over answer quality and computation cost. We instantiate this with a continuous GFlowNet using stochastic latent transitions. To handle sparse answer supervision, we introduce an Entropy-Weighted Subtrajectory Balance objective for intermediate rewards and a reference-prior regularizer to anchor exploration. Experiments under finetuning and transfer learning settings show that LTF outperforms explicit CoT and latent reasoning baselines, improving accuracy by 9.5% while reducing reasoning length by 27.2% on average compared with strong latent reasoning baselines.