The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models

Michael Rizvi-Martel; Guillaume Rabusseau; Marius Mosbach

cs.CLApr 7, 2026

The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models

Authors: Michael Rizvi-Martel, Guillaume Rabusseau, Marius Mosbach

Organizations: Mila – Quebec AI Institute · Universit´e de Montr´eal · McGill University

Abstract

Latent reasoning via continuous chain-of-thoughts (Latent CoT) has emerged as a promising alternative to discrete CoT reasoning. Operating in continuous space increases expressivity and has been hypothesized to enable superposition: the ability to maintain multiple candidate solutions simultaneously within a single representation. Despite theoretical arguments, it remains unclear whether language models actually leverage superposition when reasoning using latent CoTs. We investigate this question across three regimes: a training-free regime that constructs latent thoughts as convex combinations of token embeddings, a fine-tuned regime where a base model is adapted to produce latent thoughts, and a from-scratch regime where a model is trained entirely with latent thoughts to solve a given task. Using Logit Lens and entity-level probing to analyze internal representations, we find that only models trained from scratch exhibit signs of using superposition. In the training-free and fine-tuned regimes, we find that the superposition either collapses or is not used at all, with models discovering shortcut solutions instead. We argue that this is due to two complementary phenomena: i) pretraining on natural language data biases models to commit to a token in the last layers ii) capacity has a huge effect on which solutions a model favors. Together, our results offer a unified explanation for when and why superposition arises in continuous chain-of-thought reasoning, and identify the conditions under which it collapses.

Figures & tables

Figure 1: Two latent CoT approaches. Left: Coconut feeds the last hidden state directly back as the next input embedding, forming a recurrent loop in continuous space. Right: Soft Thinking computes a probability distribution over the vocabulary and forms the next input as a weighted sum of token embeddings. Both methods replace discrete reasoning tokens with continuous representations, but differ in how these representations are constructed.

Panel of Figure 2 — Figure 2: Superposition collapses early in the forward pass of QwQ-32B. (a) Shannon entropy shows identical patterns for Soft Thinking (orange) and discrete CoT (blue), both converging to near-zero entropy at the same rate. (b) KL divergence drops to $\sim 10^{-4}$ in middle layers, showing that Soft Thinking tokens become functionally identical to discrete tokens within the first few layers. The uncertainty in Soft Thinking token embeddings does not propagate through the network. The gray line represents the KL with a uniform distribution over the vocabulary as baseline.

Metric	Qwen2-1.5B	QwQ-32B
Cosine similarity	$0.998\pm 0.013$	$0.996\pm 0.025$
Mixing weight entropy (nats)	$0.10\pm 0.26$	$0.18\pm 0.34$

Table 1: Summary of Soft Thinking token intervention metrics. Cosine similarity is averaged across all layers, steps, and problems. Mixing entropy is the Shannon entropy of the convex combination weights

{\color[rgb]{0,0.4375,0.7539}p}_{t}

used to form each soft token.

Figure 3: Step-aware entity belief for fine-tuned GPT-2 on 5-step ProsQA examples (normalized among all entities). Coconut (left) : target entity dominates from step 0 onward with no progression. CoT (right) : correct-next entity dominates early steps, target takes over at the final step, the expected pattern for genuine multi-hop reasoning.

Table 2: ProsQA accuracy for fine-tuned and from-scratch models. (a) reports the no-latent evaluation across models. (b) reports it across model depth.

Figure 4: Step-aware entity belief for from-scratch 2-layer models on 4-step ProsQA examples. From Scratch (left) : correct-next and wrong-neighbor entities dominate intermediate steps, with belief evolving across the reasoning trace. CoT

\to

Coconut (right) : similar pattern with slightly more concentrated belief on correct-next entities. We show 4-step examples here and 5-step examples for the fine-tuned model in Figure 3 ; the qualitative pattern is consistent across hop counts (see Section E.1 ).

Panel of Figure 5 — Figure 5: Entropy of synthetic uniform superpositions for Pythia-1B, where $k$ tokens are sampled uniformly at random and combined with equal weights ( $1/k$ ). Left : entropy across layers of the fully pretrained model collapses at the final layers regardless of $k$ ; the dashed line shows a randomly initialized model, whose entropy stays near the maximum. Right : final-layer entropy across pretraining checkpoints, showing that the collapse emerges gradually as the number of training tokens increases. Shaded regions show $\pm 1$ std.

Model	Params	w/ latents	w/o latents
2L-768d	14.97M	$96.0\pm 0.9$	$30.9\pm 8.4$
4L-544d	14.78M	$91.8\pm 4.9$	$39.7\pm 20.8$
8L-384d	14.61M	$81.6\pm 4.3$	$73.2\pm 1.4$
12L-320d	15.14M	$72.4\pm 7.1$	$70.0\pm 6.3$

Table 3: Parameter-matched from-scratch Coconut on ProsQA (

{\sim}15

M parameters). Starting from the 2-layer model, depth is increased while the embedding width is shrunk to keep the parameter count fixed. Accuracies (%) are reported with and without latent tokens as mean

\pm

std over 3 seeds.

	2L		4L		8L		12L
$d_{\text{embd}}$	w/	w/o	w/	w/o	w/	w/o	w/	w/o
8	40.8	40.6	35.3	35.1	24.6	24.8	34.8	33.9
16	57.5	54.9	55.6	55.4	55.4	55.4	58.9	58.7
32	58.9	57.8	53.9	53.2	56.8	58.2	61.6	60.1
64	64.9	62.3	63.2	62.5	67.3	64.4	68.3	65.4

Table 4: Test accuracy (%) with and without latent tokens across depths, for sub-threshold embedding sizes (

d_{\text{embd}}\leq 64

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Symbol	Value
Temperature	$\tau$	0.6
Top- $k$ (sampling)	$k$	30
Soft top- $k$ (embedding mix)	$k_{\mathrm{soft}}$	15
Weighting scheme		Softmax
Max new tokens	$T_{\mathrm{max}}$	2048

Appendix

Table 5: Soft Thinking decoding hyperparameters.

Experiment	Model	GPUs	RAM	Wall Time
Comparative (Logit Lens)	Qwen2-1.5B	$1\times$ L40S	32 GB	$<1$ h
Comparative (Logit Lens)	QwQ-32B	$4\times$ L40S	128 GB	$<2$ h
Convergence	Qwen2-1.5B	$1\times$ L40S	32 GB	$<1$ h
Convergence	QwQ-32B	$4\times$ L40S	128 GB	$<3$ h
Benchmark (MATH500)	QwQ-32B	$4\times$ L40S	128 GB	varies
Benchmark (AIME2024)	QwQ-32B	$4\times$ L40S	128 GB	varies

Appendix

Table 6: Compute resources per experiment.

Figure 6: Top 3 tokens at the output layer for time steps with largest (top) and smallest (bottom) KL divergence between soft and argmax representations in QwQ-32B . Each column represents a problem instance.

Figure 7: Top 3 predicted tokens at the output layer for time steps with largest (top) and smallest (bottom) KL divergence between soft and argmax representations in Qwen2-1.5B . Each column represents a problem instance.

Figure 8: Dataset-averaged intervention metrics for QwQ-32B on MATH500 (N=500). Left to right: KL divergence, cosine similarity, absolute entropy difference. Each cell is the mean over all problems at that (layer, relative position) bin.

Figure 9: Dataset-averaged intervention metrics for QwQ-32B on AIME 2024 (N=30).

Figure 10: Dataset-averaged intervention metrics for Qwen2-1.5B on MATH500 (N=500).

Figure 11: Dataset-averaged intervention metrics for Qwen2-1.5B on GSM8K (N=500).

Figure 12: Dataset-averaged intervention metrics for Qwen2-1.5B on AIME 2024 (N=30).

Figure 13: Dataset-averaged intervention metrics for DeepSeek-R1-Distill-Llama-70B on MATH500 (N=500).

Figure 14: Dataset-averaged intervention metrics for DeepSeek-R1-Distill-Llama-70B on AIME 2024 (N=30).

Panel of Figure 15 — Figure 15: QwQ-32B on AIME2024 (N=30). Entropy and KL divergence by layer.

Panel of Figure 16 — Figure 16: Qwen2-1.5B on MATH500 (N=500). Entropy and KL divergence by layer.

Panel of Figure 17 — Figure 17: Qwen2-1.5B on GSM8K (N=500). Entropy and KL divergence by layer.

Panel of Figure 18 — Figure 18: Qwen2-1.5B on AIME2024 (N=30). Entropy and KL divergence by layer.

Panel of Figure 19 — Figure 19: DeepSeek-R1-Distill-Llama-70B on MATH500 (N=500). Entropy and KL divergence by layer. Logit Lens is applied at 5 evenly spaced layers $\{0,20,40,60,79\}$ of the 80-layer model.

Panel of Figure 20 — Figure 20: DeepSeek-R1-Distill-Llama-70B on AIME2024 (N=30). Entropy and KL divergence by layer. Same 5-layer probe as Figure 19 ; KL remains near zero from layer 20 onward.

Figure 21: Logit Lens entropy at reasoning positions for fine-tuned GPT-2 (left, 12 layers) and a from-scratch 2-layer model (right). In the fine-tuned model, Coconut latent positions maintain uniformly low entropy across all layers, while CoT positions show high early-layer entropy that resolves by layers 8–9. In the from-scratch model, Coconut latent positions retain higher entropy than their CoT counterparts, consistent with the from-scratch model maintaining richer latent representations. Shaded regions denote

\pm 1

standard deviation across examples.

Panel of Figure 22 — Figure 22: QwQ-32B, uniform weighting $k=3$ . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right). Despite uniform weights giving equal mass to 3 tokens, representations collapse to near-argmax in middle layers on both datasets.

Panel of Figure 23 — Figure 23: QwQ-32B, uniform weighting $k=10$ . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right).

Panel of Figure 24 — Figure 24: QwQ-32B, uniform weighting $k=15$ . Matching the top- $k$ of the standard softmax experiments, uniform weighting still collapses in middle layers. The entropy profiles are indistinguishable from discrete CoT.

Panel of Figure 25 — Figure 25: Qwen2-1.5B, uniform weighting $k=3$ . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right). Collapse is consistent with the 32B findings.

Panel of Figure 26 — Figure 26: Qwen2-1.5B, uniform weighting $k=10$ . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right).

Panel of Figure 27 — Figure 27: Qwen2-1.5B, uniform weighting $k=15$ . Entropy profiles (top) and KL divergence (bottom) for AIME 2024 (left) and MATH500 (right). Results are consistent across both model scales: uniform superposition collapses regardless of $k$ .

Run	Decoding	max_topk	Cold Stop	Accuracy (%)	Avg Tokens
1	Discrete	–	0.0 (none)	97.08	4,326
2	Discrete	–	0.1	97.08	4,326
3	Discrete	–	0.2	97.19	4,307
4	Softmax	10	0.0 (none)	96.47	4,222
5	Softmax	10	0.1	96.84	4,056
6	Softmax	15	0.01	96.84	4,044

Appendix

Table 7: MATH500 results with QwQ-32B (N=500). Discrete decoding uses standard greedy/sampling; Softmax and Uniform refer to the weighting scheme used to construct Soft Thinking embeddings.

Run	Decoding	max_topk	Cold Stop	Accuracy (%)	Avg Tokens
1	Discrete	–	0.0 (none)	77.29	13,445
2	Discrete	–	0.1	77.29	13,445
3	Discrete	–	0.2	77.29	13,445
4	Softmax	15	0.01	75.83	12,445
5	Softmax	15	0.1	76.25	11,818

Appendix

Table 8: AIME2024 results with QwQ-32B (N=30).

Parameter	CoT baseline	Coconut
Base model	GPT-2 (124M)	GPT-2 (124M)
Learning rate	$10^{-4}$	$10^{-4}$
Optimizer	AdamW	AdamW
Weight decay	0.01	0.01
Batch size (per device)	16	16
Gradient accumulation steps	2	2

Appendix

Table 9: Coconut training hyperparameters on ProsQA.

Figure 28: Normalized entity probability mass at each reasoning step for 4-step ProsQA examples (fine-tuned Coconut). Each stacked area shows the fraction of probability assigned to four entity categories: correct next hop (blue), wrong graph neighbors (orange), target/final answer (red), and other entities (gray). In the CoT model, probability mass shifts progressively toward the correct next entity at each step, consistent with step-by-step chain traversal. In the Coconut model, the target entity dominates from the first reasoning step onward, indicating that the model commits to the final answer without tracking intermediate hops.

Figure 29: Step-aware entity belief across model depths for 3-step ProsQA examples (from-scratch Coconut). Each panel corresponds to a model trained from random initialization at a different depth (2L, 4L, 8L, 12L). Shallow models (2L, 4L) exhibit belief evolution across latent steps, with probability mass shifting toward the correct next entity at intermediate positions. Deeper models (8L, 12L) show no such progression; the target entity appears early without intermediate tracking.

Panel of Figure 30 — Figure 30: Step-aware entity belief for the 3.4% of ProsQA test examples (17/500) that the Coconut model answers correctly with latent tokens but incorrectly without them. In the Coconut panel, the “other” category dominates ( ${\sim}70\%$ ), with the target receiving only ${\sim}25{-}35\%$ of the probability mass; the model does not confidently commit to the final answer from the question embedding alone on these examples, unlike the general population ( Figure 28 ). In contrast, the CoT model still performs step-by-step chain traversal on the same examples. This suggests that the latent tokens may be load-bearing for a small subset of examples where the question embedding does not suffice.

Panel of Figure 31 — Figure 31: Normalized entity probability mass at each reasoning step on ProntoQA (fine-tuned Coconut). In the Coconut model, the “other” category dominates throughout ( ${\sim}80\%$ ), with correct next, wrong neighbors, and target each receiving minimal probability mass. In contrast, the CoT model tracks the correct next entity at each intermediate step before transitioning to the target at the final step.

Figure 32: Normalized probability of the correct label (“True” or “False”) across latent positions for the Coconut model on ProntoQA. At the pre-thinking position the model assigns

{\sim}93\%

of the probability mass to the correct label; this drops to

{\sim}60\%

during latent reasoning (T1–T5) before recovering at the post-thinking position. The dip during latent positions is consistent with the model redistributing probability mass without performing productive intermediate computation. Probing for these labels is more consistent with the ProntoQA task as the model must return a binary “True or False” response, as opposed to returning the entity name as is the case in ProsQA.

Figure 33: Mean

L_{2}

gradient norms per parameter group during Coconut fine-tuning on ProsQA (GPT-2). Each panel shows one parameter group: word token embeddings (wte + lm_head), attention, MLP, LayerNorm, and positional embeddings (wpe). Alternating shaded bands denote the five-epoch training stages of the Coconut curriculum. Note that gradient magnitudes remain non-trivial across all groups and stages, indicating that the model parameters are being actively updated throughout training; however, as shown in Section 5 , this training does not yield latent representations that participate in multi-step reasoning.

Figure 34: Logit Lens entropy at reasoning positions for fine-tuned GPT-2 (left) and a from-scratch 2-layer model (right). In the fine-tuned model, Coconut latent positions maintain near-zero entropy across all layers, consistent with early commitment to a single token. In the from-scratch model, this pattern reverses: Coconut latent positions retain higher entropy than CoT, suggesting richer latent representations. Shaded regions denote

\pm 1

standard deviation.

Model	CoT	Coconut	Coconut (no latents)	Drop
GPT-2 (124M)	85.3	99.0	99.0	$-0.0$
SmolLM2-135M	72.7	93.3	92.3	$-1.0$
SmolLM2-360M	85.0	98.7	98.0	$-0.7$
SmolLM2-1.7B	98.3	100.0	100.0	$-0.0$

Appendix

Table 10: Counterfactual no-latent evaluation on ProsQA across independently trained models. Coconut (no latents) removes all latent tokens and scores both candidate completions; Drop

=

Coconut (no latents)

-

Coconut.

	2-Layer			4-Layer
$d_{\text{embd}}$	w/ latents	w/o latents	gap	w/ latents	w/o latents	gap
8	40.8	40.6	$+$ 0.2	35.3	35.1	$+$ 0.2
16	57.5	54.9	$+$ 2.6	55.6	55.4	$+$ 0.2
32	58.9	57.8	$+$ 1.1	53.9	53.2	$+$ 0.7
64	64.9	62.3	$+$ 2.6	63.2	62.5	$+$ 0.7
128	67.1	60.6	$+$ 6.5	61.3	60.4	$+$ 0.9

Appendix

Table 11: Test accuracy (%) with and without latent tokens across embedding dimensions, for 2-layer and 4-layer from-scratch Coconut models on ProsQA. The gap (w/

-

w/o) measures reliance on latent reasoning; a large gap indicates the model genuinely uses latent tokens. Graph width

\approx 28

nodes; the theoretical lower bound is

\Omega(28)

Depth	w/ latents	w/o latents	gap
2L	$96.0\pm 0.9$	$30.9\pm 8.4$	$65.1\pm 9.0$
4L	$96.9\pm 1.2$	$33.2\pm 25.0$	$63.7\pm 23.7$
8L	$83.4\pm 2.5$	$67.9\pm 5.5$	$15.4\pm 3.1$
12L	$71.7\pm 8.8$	$54.4\pm 23.4$	$17.3\pm 26.8$

Appendix

Table 12: From-scratch Coconut on ProsQA (

d_{\text{embd}}=768

): accuracy (%) with and without latent tokens, across 3 independent seeds. Gap = w/

-

w/o. Shallow models (2L, 4L) show a large, consistent gap; deeper models show a smaller but more variable gap, consistent with the depth-dependent shortcut pattern discussed in the main text.

Continuous Latent Reasoning

Continuous latent reasoning studies how models can compute through continuous hidden states instead of spelling out every reasoning step.

See all 80 papers

SuperThoughts: Reasoning Tokens in Superposition
Jun 11, 2026Zheyang Xiong, Shivam Garg, Max Yu +4Multi-Token Prediction LLM Inference Acceleration
mMicrosoft Research · iIndependent · pPrinceton University +2
Latent Thought Flow: Efficient Latent Reasoning in Large Language Models
Jun 15, 2026Xiandong Zou, Jing Huang, Jianshu Li +1RL for Language Model Reasoning Efficient Language Model Reasoning
Singapore Management University · Ant Group
Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs
Sep 1, 2026Zhaoliang Chen, Jie FuEfficient Language Model Reasoning Continuous Latent Reasoning
Emory University · IQuest Research

Jun 11, 2026J/K move · Enter open · S save

SuperThoughts: Reasoning Tokens in Superposition

Zheyang Xiong, Shivam Garg, Max Yu, Vaishnavi Shrivastava+3

mMicrosoft Research · iIndependent · pPrinceton University+2

Multi-Token Prediction LLM Inference Acceleration Mathematical Reasoning CoT Compression LLM Reasoning

Open paper AskPDF arXiv

Long Chain-of-Thought (CoT) reasoning improves LLM problem-solving but is computationally expensive due to sequential token generation. While recent works explore reasoning in continuous latent spaces to bypass discrete token generation, they often struggle with training stability and fail to scale to complex, long-horizon tasks due to lack of supervision signal. We propose SuperThoughts, which compresses pairs of consecutive CoT tokens into single latent representations and decodes two tokens per step via a lightweight Multi-Token Prediction (MTP) module. This preserves discrete token supervision at training time while doubling throughput at inference time. We finetune Qwen2.5-Math-1.5B-Instruct, Qwen2.5-Math-7B-Instruct, Qwen2.5-Math-14B-Instruct, and evaluate on MATH500, AMC, OlympiadBench, and GPQA-Diamond. With a confidence-based adaptive mechanism that falls back to standard decoding when uncertain, SuperThoughts achieves

\sim

20--30% CoT length reduction while maintaining accuracy with minimal degradation (1-2 points accuracy drop on most tasks).

Abstract

Continuous Latent Reasoning

Explore similar work

SuperThoughts: Reasoning Tokens in Superposition

Latent Thought Flow: Efficient Latent Reasoning in Large Language Models

Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs