Latent reasoning via continuous chain-of-thoughts (Latent CoT) has emerged as a promising alternative to discrete CoT reasoning. Operating in continuous space increases expressivity and has been hypothesized to enable superposition: the ability to maintain multiple candidate solutions simultaneously within a single representation. Despite theoretical arguments, it remains unclear whether language models actually leverage superposition when reasoning using latent CoTs. We investigate this question across three regimes: a training-free regime that constructs latent thoughts as convex combinations of token embeddings, a fine-tuned regime where a base model is adapted to produce latent thoughts, and a from-scratch regime where a model is trained entirely with latent thoughts to solve a given task. Using Logit Lens and entity-level probing to analyze internal representations, we find that only models trained from scratch exhibit signs of using superposition. In the training-free and fine-tuned regimes, we find that the superposition either collapses or is not used at all, with models discovering shortcut solutions instead. We argue that this is due to two complementary phenomena: i) pretraining on natural language data biases models to commit to a token in the last layers ii) capacity has a huge effect on which solutions a model favors. Together, our results offer a unified explanation for when and why superposition arises in continuous chain-of-thought reasoning, and identify the conditions under which it collapses.
Figures & tables
Figure 1: Two latent CoT approaches. Left: Coconut feeds the last hidden state directly back as the next input embedding, forming a recurrent loop in continuous space. Right: Soft Thinking computes a probability distribution over the vocabulary and forms the next input as a weighted sum of token embeddings. Both methods replace discrete reasoning tokens with continuous representations, but differ in how these representations are constructed.
Figure 2: Superposition collapses early in the forward pass of QwQ-32B. (a) Shannon entropy shows identical patterns for Soft Thinking (orange) and discrete CoT (blue), both converging to near-zero entropy at the same rate. (b) KL divergence drops to ∼10−4 in middle layers, showing that Soft Thinking tokens become functionally identical to discrete tokens within the first few layers. The uncertainty in Soft Thinking token embeddings does not propagate through the network. The gray line represents the KL with a uniform distribution over the vocabulary as baseline.
Metric
Qwen2-1.5B
QwQ-32B
Cosine similarity
0.998±0.013
0.996±0.025
Mixing weight entropy (nats)
0.10±0.26
0.18±0.34
Table 1: Summary of Soft Thinking token intervention metrics. Cosine similarity is averaged across all layers, steps, and problems. Mixing entropy is the Shannon entropy of the convex combination weights {\color[rgb]{0,0.4375,0.7539}p}_{t} used to form each soft token.
Figure 3: Step-aware entity belief for fine-tuned GPT-2 on 5-step ProsQA examples (normalized among all entities). Coconut (left) : target entity dominates from step 0 onward with no progression. CoT (right) : correct-next entity dominates early steps, target takes over at the final step, the expected pattern for genuine multi-hop reasoning.
Table 2: ProsQA accuracy for fine-tuned and from-scratch models. (a) reports the no-latent evaluation across models. (b) reports it across model depth.
Figure 4: Step-aware entity belief for from-scratch 2-layer models on 4-step ProsQA examples. From Scratch (left) : correct-next and wrong-neighbor entities dominate intermediate steps, with belief evolving across the reasoning trace. CoT → Coconut (right) : similar pattern with slightly more concentrated belief on correct-next entities. We show 4-step examples here and 5-step examples for the fine-tuned model in Figure 3 ; the qualitative pattern is consistent across hop counts (see Section E.1 ).
Figure 5: Entropy of synthetic uniform superpositions for Pythia-1B, where k tokens are sampled uniformly at random and combined with equal weights ( 1/k ). Left : entropy across layers of the fully pretrained model collapses at the final layers regardless of k ; the dashed line shows a randomly initialized model, whose entropy stays near the maximum. Right : final-layer entropy across pretraining checkpoints, showing that the collapse emerges gradually as the number of training tokens increases. Shaded regions show ±1 std.
Model
Params
w/ latents
w/o latents
2L-768d
14.97M
96.0±0.9
30.9±8.4
4L-544d
14.78M
91.8±4.9
39.7±20.8
8L-384d
14.61M
81.6±4.3
73.2±1.4
12L-320d
15.14M
72.4±7.1
70.0±6.3
Table 3: Parameter-matched from-scratch Coconut on ProsQA ( ∼15 M parameters). Starting from the 2-layer model, depth is increased while the embedding width is shrunk to keep the parameter count fixed. Accuracies (%) are reported with and without latent tokens as mean ± std over 3 seeds.
2L
4L
8L
12L
dembd
w/
w/o
w/
w/o
w/
w/o
w/
w/o
8
40.8
40.6
35.3
35.1
24.6
24.8
34.8
33.9
16
57.5
54.9
55.6
55.4
55.4
55.4
58.9
58.7
32
58.9
57.8
53.9
53.2
56.8
58.2
61.6
60.1
64
64.9
62.3
63.2
62.5
67.3
64.4
68.3
65.4
Table 4: Test accuracy (%) with and without latent tokens across depths, for sub-threshold embedding sizes ( dembd≤64 ).
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Symbol
Value
Temperature
τ
0.6
Top- k (sampling)
k
30
Soft top- k (embedding mix)
ksoft
15
Weighting scheme
Softmax
Max new tokens
Tmax
2048
Appendix
Table 5: Soft Thinking decoding hyperparameters.
Experiment
Model
GPUs
RAM
Wall Time
Comparative (Logit Lens)
Qwen2-1.5B
1× L40S
32 GB
<1 h
Comparative (Logit Lens)
QwQ-32B
4× L40S
128 GB
<2 h
Convergence
Qwen2-1.5B
1× L40S
32 GB
<1 h
Convergence
QwQ-32B
4× L40S
128 GB
<3 h
Benchmark (MATH500)
QwQ-32B
4× L40S
128 GB
varies
Benchmark (AIME2024)
QwQ-32B
4× L40S
128 GB
varies
Appendix
Table 6: Compute resources per experiment.
Figure 6: Top 3 tokens at the output layer for time steps with largest (top) and smallest (bottom) KL divergence between soft and argmax representations in QwQ-32B . Each column represents a problem instance.
Figure 7: Top 3 predicted tokens at the output layer for time steps with largest (top) and smallest (bottom) KL divergence between soft and argmax representations in Qwen2-1.5B . Each column represents a problem instance.
Figure 8: Dataset-averaged intervention metrics for QwQ-32B on MATH500 (N=500). Left to right: KL divergence, cosine similarity, absolute entropy difference. Each cell is the mean over all problems at that (layer, relative position) bin.
Figure 9: Dataset-averaged intervention metrics for QwQ-32B on AIME 2024 (N=30).
Figure 10: Dataset-averaged intervention metrics for Qwen2-1.5B on MATH500 (N=500).
Figure 11: Dataset-averaged intervention metrics for Qwen2-1.5B on GSM8K (N=500).
Figure 12: Dataset-averaged intervention metrics for Qwen2-1.5B on AIME 2024 (N=30).
Figure 13: Dataset-averaged intervention metrics for DeepSeek-R1-Distill-Llama-70B on MATH500 (N=500).
Figure 14: Dataset-averaged intervention metrics for DeepSeek-R1-Distill-Llama-70B on AIME 2024 (N=30).
Figure 15: QwQ-32B on AIME2024 (N=30). Entropy and KL divergence by layer.
Figure 16: Qwen2-1.5B on MATH500 (N=500). Entropy and KL divergence by layer.
Figure 17: Qwen2-1.5B on GSM8K (N=500). Entropy and KL divergence by layer.
Figure 18: Qwen2-1.5B on AIME2024 (N=30). Entropy and KL divergence by layer.
Figure 19: DeepSeek-R1-Distill-Llama-70B on MATH500 (N=500). Entropy and KL divergence by layer. Logit Lens is applied at 5 evenly spaced layers {0,20,40,60,79} of the 80-layer model.
Figure 20: DeepSeek-R1-Distill-Llama-70B on AIME2024 (N=30). Entropy and KL divergence by layer. Same 5-layer probe as Figure 19 ; KL remains near zero from layer 20 onward.
Figure 21: Logit Lens entropy at reasoning positions for fine-tuned GPT-2 (left, 12 layers) and a from-scratch 2-layer model (right). In the fine-tuned model, Coconut latent positions maintain uniformly low entropy across all layers, while CoT positions show high early-layer entropy that resolves by layers 8–9. In the from-scratch model, Coconut latent positions retain higher entropy than their CoT counterparts, consistent with the from-scratch model maintaining richer latent representations. Shaded regions denote ±1 standard deviation across examples.
Figure 22: QwQ-32B, uniform weighting k=3 . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right). Despite uniform weights giving equal mass to 3 tokens, representations collapse to near-argmax in middle layers on both datasets.
Figure 23: QwQ-32B, uniform weighting k=10 . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right).
Figure 24: QwQ-32B, uniform weighting k=15 . Matching the top- k of the standard softmax experiments, uniform weighting still collapses in middle layers. The entropy profiles are indistinguishable from discrete CoT.
Figure 25: Qwen2-1.5B, uniform weighting k=3 . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right). Collapse is consistent with the 32B findings.
Figure 26: Qwen2-1.5B, uniform weighting k=10 . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right).
Figure 27: Qwen2-1.5B, uniform weighting k=15 . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right). Results are consistent across both model scales: uniform superposition collapses regardless of k .
Run
Decoding
max_topk
Cold Stop
Accuracy (%)
Avg Tokens
1
Discrete
–
0.0 (none)
97.08
4,326
2
Discrete
–
0.1
97.08
4,326
3
Discrete
–
0.2
97.19
4,307
4
Softmax
10
0.0 (none)
96.47
4,222
5
Softmax
10
0.1
96.84
4,056
6
Softmax
15
0.01
96.84
4,044
Appendix
Table 7: MATH500 results with QwQ-32B (N=500). Discrete decoding uses standard greedy/sampling; Softmax and Uniform refer to the weighting scheme used to construct Soft Thinking embeddings.
Run
Decoding
max_topk
Cold Stop
Accuracy (%)
Avg Tokens
1
Discrete
–
0.0 (none)
77.29
13,445
2
Discrete
–
0.1
77.29
13,445
3
Discrete
–
0.2
77.29
13,445
4
Softmax
15
0.01
75.83
12,445
5
Softmax
15
0.1
76.25
11,818
Appendix
Table 8: AIME2024 results with QwQ-32B (N=30).
Parameter
CoT baseline
Coconut
Base model
GPT-2 (124M)
GPT-2 (124M)
Learning rate
10−4
10−4
Optimizer
AdamW
AdamW
Weight decay
0.01
0.01
Batch size (per device)
16
16
Gradient accumulation steps
2
2
Appendix
Table 9: Coconut training hyperparameters on ProsQA.
Figure 28: Normalized entity probability mass at each reasoning step for 4-step ProsQA examples (fine-tuned Coconut). Each stacked area shows the fraction of probability assigned to four entity categories: correct next hop (blue), wrong graph neighbors (orange), target/final answer (red), and other entities (gray). In the CoT model, probability mass shifts progressively toward the correct next entity at each step, consistent with step-by-step chain traversal. In the Coconut model, the target entity dominates from the first reasoning step onward, indicating that the model commits to the final answer without tracking intermediate hops.
Figure 29: Step-aware entity belief across model depths for 3-step ProsQA examples (from-scratch Coconut). Each panel corresponds to a model trained from random initialization at a different depth (2L, 4L, 8L, 12L). Shallow models (2L, 4L) exhibit belief evolution across latent steps, with probability mass shifting toward the correct next entity at intermediate positions. Deeper models (8L, 12L) show no such progression; the target entity appears early without intermediate tracking.
Figure 30: Step-aware entity belief for the 3.4% of ProsQA test examples (17/500) that the Coconut model answers correctly with latent tokens but incorrectly without them. In the Coconut panel, the “other” category dominates ( ∼70% ), with the target receiving only ∼25−35% of the probability mass; the model does not confidently commit to the final answer from the question embedding alone on these examples, unlike the general population ( Figure 28 ). In contrast, the CoT model still performs step-by-step chain traversal on the same examples. This suggests that the latent tokens may be load-bearing for a small subset of examples where the question embedding does not suffice.
Figure 31: Normalized entity probability mass at each reasoning step on ProntoQA (fine-tuned Coconut). In the Coconut model, the “other” category dominates throughout ( ∼80% ), with correct next, wrong neighbors, and target each receiving minimal probability mass. In contrast, the CoT model tracks the correct next entity at each intermediate step before transitioning to the target at the final step.
Figure 32: Normalized probability of the correct label (“True” or “False”) across latent positions for the Coconut model on ProntoQA. At the pre-thinking position the model assigns ∼93% of the probability mass to the correct label; this drops to ∼60% during latent reasoning (T1–T5) before recovering at the post-thinking position. The dip during latent positions is consistent with the model redistributing probability mass without performing productive intermediate computation. Probing for these labels is more consistent with the ProntoQA task as the model must return a binary “True or False” response, as opposed to returning the entity name as is the case in ProsQA.
Figure 33: Mean L2 gradient norms per parameter group during Coconut fine-tuning on ProsQA (GPT-2). Each panel shows one parameter group: word token embeddings (wte + lm_head), attention, MLP, LayerNorm, and positional embeddings (wpe). Alternating shaded bands denote the five-epoch training stages of the Coconut curriculum. Note that gradient magnitudes remain non-trivial across all groups and stages, indicating that the model parameters are being actively updated throughout training; however, as shown in Section 5 , this training does not yield latent representations that participate in multi-step reasoning.
Figure 34: Logit Lens entropy at reasoning positions for fine-tuned GPT-2 (left) and a from-scratch 2-layer model (right). In the fine-tuned model, Coconut latent positions maintain near-zero entropy across all layers, consistent with early commitment to a single token. In the from-scratch model, this pattern reverses: Coconut latent positions retain higher entropy than CoT, suggesting richer latent representations. Shaded regions denote ±1 standard deviation.
Model
CoT
Coconut
Coconut (no latents)
Drop
GPT-2 (124M)
85.3
99.0
99.0
−0.0
SmolLM2-135M
72.7
93.3
92.3
−1.0
SmolLM2-360M
85.0
98.7
98.0
−0.7
SmolLM2-1.7B
98.3
100.0
100.0
−0.0
Appendix
Table 10: Counterfactual no-latent evaluation on ProsQA across independently trained models. Coconut (no latents) removes all latent tokens and scores both candidate completions; Drop = Coconut (no latents) − Coconut.
2-Layer
4-Layer
dembd
w/ latents
w/o latents
gap
w/ latents
w/o latents
gap
8
40.8
40.6
+ 0.2
35.3
35.1
+ 0.2
16
57.5
54.9
+ 2.6
55.6
55.4
+ 0.2
32
58.9
57.8
+ 1.1
53.9
53.2
+ 0.7
64
64.9
62.3
+ 2.6
63.2
62.5
+ 0.7
128
67.1
60.6
+ 6.5
61.3
60.4
+ 0.9
Appendix
Table 11: Test accuracy (%) with and without latent tokens across embedding dimensions, for 2-layer and 4-layer from-scratch Coconut models on ProsQA. The gap (w/ − w/o) measures reliance on latent reasoning; a large gap indicates the model genuinely uses latent tokens. Graph width ≈28 nodes; the theoretical lower bound is Ω(28) .
Depth
w/ latents
w/o latents
gap
2L
96.0±0.9
30.9±8.4
65.1±9.0
4L
96.9±1.2
33.2±25.0
63.7±23.7
8L
83.4±2.5
67.9±5.5
15.4±3.1
12L
71.7±8.8
54.4±23.4
17.3±26.8
Appendix
Table 12: From-scratch Coconut on ProsQA ( dembd=768 ): accuracy (%) with and without latent tokens, across 3 independent seeds. Gap = w/ − w/o. Shallow models (2L, 4L) show a large, consistent gap; deeper models show a smaller but more variable gap, consistent with the depth-dependent shortcut pattern discussed in the main text.
Long Chain-of-Thought (CoT) reasoning improves LLM problem-solving but is computationally expensive due to sequential token generation. While recent works explore reasoning in continuous latent spaces to bypass discrete token generation, they often struggle with training stability and fail to scale to complex, long-horizon tasks due to lack of supervision signal. We propose SuperThoughts, which compresses pairs of consecutive CoT tokens into single latent representations and decodes two tokens per step via a lightweight Multi-Token Prediction (MTP) module. This preserves discrete token supervision at training time while doubling throughput at inference time. We finetune Qwen2.5-Math-1.5B-Instruct, Qwen2.5-Math-7B-Instruct, Qwen2.5-Math-14B-Instruct, and evaluate on MATH500, AMC, OlympiadBench, and GPQA-Diamond. With a confidence-based adaptive mechanism that falls back to standard decoding when uncertain, SuperThoughts achieves ∼20--30% CoT length reduction while maintaining accuracy with minimal degradation (1-2 points accuracy drop on most tasks).
Zheyang Xiong, Shivam Garg, Max Yu +4
mMicrosoft Research · iIndependent · pPrinceton University +2
Large Language Models (LLMs) increasingly rely on intermediate reasoning, yet explicit Chain-of-Thought (CoT) suffers from a linguistic space bottleneck: each thought must be decoded into tokens, causing high inference overhead. Latent reasoning moves deliberation into continuous space, but existing methods mostly learn deterministic or reward-maximizing paths, lacking a principled way to allocate probability across trajectories with different correctness and costs. We propose Latent Thought Flow (LTF), which models reasoning as variable-length continuous trajectories and trains a sampler to match a reward-induced posterior over answer quality and computation cost. We instantiate this with a continuous GFlowNet using stochastic latent transitions. To handle sparse answer supervision, we introduce an Entropy-Weighted Subtrajectory Balance objective for intermediate rewards and a reference-prior regularizer to anchor exploration. Experiments under finetuning and transfer learning settings show that LTF outperforms explicit CoT and latent reasoning baselines, improving accuracy by 9.5% while reducing reasoning length by 27.2% on average compared with strong latent reasoning baselines.
Chain-of-thought reasoning unfolds in discrete token space: each step is committed as text, errors propagate, and eliciting good traces presupposes traces to imitate. Reasoning instead in a model's continuous representation space - where intermediate states are vectors rather than words - sidesteps these constraints, but leaves open how those latent states should be computed. We approach this along two axes. First, we keep a large language model (LLM) frozen and use it for what it is already good at - modeling and decoding sequences - while a small auxiliary network supplies continuous latent thoughts as input. Second, we produce those latents by recurrence: a tiny recurrent reasoner refines them over many steps, decoupling the depth of computation from the size of the model, so that the latents are a product of iterative processing rather than a single forward pass. We instantiate this as Latent Recurrent Thoughts (LRT): a task-dedicated proposer supplies base latents, a recurrent reasoner refines them through bounded residual corrections, and the frozen LLM decodes the answer. On symbolic reasoning with answer supervision but no reasoning traces (Countdown-4, Sudoku) and on natural-language reasoning (HumanEval, MBPP, StrategyQA), LRT substantially outperforms prior frozen-decoder continuous-space reasoning methods under an identical decoder, prompt, data, and training budget, and outperforms non-thinking-mode chain-of-thought prompting on the same backbone at a small fraction of its inference compute.