Sep-Latent: Separator Tokens as Natural Compression Anchors for Implicit Reasoning

Yinlong Xu; Jianfeng Huang; Zhengdong Zhou; Mingjie Lin; Fuyu Lv; Dan Ou; Hongxia Xu; Jian Wu

cs.AIOct 16, 2026

Sep-Latent: Separator Tokens as Natural Compression Anchors for Implicit Reasoning

Authors: Yinlong Xu, Jianfeng Huang, Zhengdong Zhou, Mingjie Lin, Fuyu Lv, Dan Ou, Hongxia Xu, Jian Wu

Organizations: College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Alibaba Group, Hangzhou, China

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning through explicit Chain-of-Thought (CoT), but the associated computational overhead and token verbosity remain significant barriers to efficiency. While implicit reasoning in continuous latent spaces offers a promising alternative, existing methods like Coconut and CODI often rely on fixed-length latent sequences and suffer from representational collapse. In this paper, we propose Sep-Latent, an adaptive implicit reasoning framework that utilizes separator tokens as structural semantic anchors. By introducing an Attention-Decay mechanism, we force the model to aggregate and compress reasoning logic into these anchors during training. We then align explicit and implicit pathways through multi-level distillation across all layers to ensure logical consistency. Crucially, Sep-Latent enables dynamic reasoning depth by adaptively terminating the latent thinking phase based on query complexity. Experimental results on the GSM8k-Aug benchmark show that Sep-Latent achieves an accuracy of 56.8% with an average of only 5.4 tokens, significantly outperforming prior fixed-length implicit models in efficiency while narrowing the performance gap with explicit CoT. Further analysis via PCA and symbolic variant testing confirms that our method ensures semantic grounding and exhibits superior robustness to perturbations.

Figures & tables

Figure 1: Language models naturally allocate strong attention to separator tokens during reasoning. Increasing attention concentration on separator tokens preserves reasoning accuracy and perplexity with negligible performance variation

Figure 2: Algorithm of Sep-Latent: As the training epochs proceed, LLM pays less and less attention to the reasoning step, and the information is gradually compressed into <sep>. It should be noted that although the figure uses two LLMS to illustrate the algorithm, there is actually one model during the actual training process.

Llama3.2-1B
Method	GSM8k-Aug		GSM-Hard		MultiArith		SVAMP		Avg.
Method	Acc. (%)	tokens	Acc. (%)	tokens	Acc. (%)	tokens	Acc. (%)	tokens	Acc. (%)	tokens
SFT-CoT	56.2	25.3	13.9	23.1	96.7	11.1	65.7	12.1	58.1	17.9
SFT-Direct	28.8	1.2	6.3	1.9	50.3	1.9	26.7	1.9	28.0	1.9
Coconut	33.2	7.2	7.0	5.9	63.3	5.9	43.7	5.9	36.8	5.9
SIM-Coconut	42.2	7.2	9.3	5.9	87.7	5.9	43.9	5.9	45.8	5.9

Table 1: Main results on Llama3.2-1B-Instruct and Qwen3-0.6B. We report accuracy (%) and token consumption on each benchmark. For Sep-Latent, we report the best performance with the

decay\_ratio=0.25

. Bold indicates the best result, while underlined values denote the second-best result.

Figure 3: The result of SFT-CoT and Sep-Latent on GSM-SYM benchmarks.

Method	GSM8k-Aug		GSM-Hard		MultiArith		SVAMP		Avg.
Method	Acc. (%)	tokens	Acc. (%)	tokens	Acc. (%)	tokens	Acc. (%)	tokens	Acc. (%)	tokens
Llama3.2-3B
SFT-CoT	71.5	27.7	17.0	22.4	98.3	11.4	71.0	12.4	64.5	18.5
SFT-Direct	38.3	1.2	9.5	1.4	88.7	1.4	52.9	1.4	47.4	1.4
CODI	60.8	8.2	14.3	11.5	98.7	11.5	73.3	11.5	61.8	10.7
SIM-CODI	62.3	8.2	14.6	11.5	98.8	11.5	74.9	11.5	62.7	10.7

Table 2: Results on Llama3.2-3B and Llama3.1-8B models. We report accuracy (%) and token consumption on each benchmark. Bold indicates the best result, while underlined values denote the second-best result.

Method	Logi	CSQA	Strategy	Avg.
SFT-CoT	39.8	68.2	41.1	49.7
SFT-Direct	46.2	74.9	63.4	61.5
CODI	44.6	72.4	61.8	59.6
SIM-CODI	45.2	73.2	63.1	60.5
Sep-Latent	47.1	73.9	62.9	61.3

Table 3: Commonsense reasoning results on Llama3.2-1B-Instruct. We report accuracy(%) on three datasets: LogiQA, CommonsenseQA and StrategyQA.

Figure 4: PCA results of token representations across different methods. Sep-Latent stays closer to the vocabulary embedding space than existing implicit reasoning methods.

Figure 5: Comparative Analysis of Training Overhead: GPU Memory vs. Training Time Marker size is proportional to model accuracy (ACC), with larger bubbles representing better performance

Figure 6: The ablation studies about Attention-Decay Mechanism

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: The reasoning steps distribution of training data. Most problems involve 1-4 steps and all the data are used for our training.

Figure 8: The baselines used in our paper.

Model	Method	Lr	Epochs
Llama 1B	SFT-CoT	$1\times 10^{-4}$	3
Llama 1B	SFT-Direct	$1\times 10^{-4}$	3
Llama 1B	Coconut	$1\times 10^{-4}$	30
Llama 1B	SIM-Coconut	$1\times 10^{-4}$	30
Llama 1B	CODI	$8\times 10^{-4}$	10
Llama 1B	SIM-CODI	$8\times 10^{-4}$	10

Appendix

Table 4: Training hyperparameters across different baselines.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 356 papers

SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models
Mar 6, 2026Yunlong Chu, Minglai Shao, Yuhang Liu +4Efficient Language Model Reasoning CoT Compression
School of New Media and Communication, Tianjin University, Tianjin, China · Independent Contributor, CA, USA · School of Computer Science and Engineering, Beihang University, Beijing, China
Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains
May 25, 2026Hui Xie, Jie Liu, Ziyue Qiao +1Efficient Language Model Reasoning LLM Inference Acceleration
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning
Sep 7, 2026Yuwen Hao, Menglin YangSpectral Regularization Continuous Latent Reasoning
The Hong Kong University of Science and Technology (Guangzhou)

Mar 6, 2026J/K move · Enter open · S save

SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models

Yunlong Chu, Minglai Shao, Yuhang Liu, Bing Hao+3

School of New Media and Communication, Tianjin University, Tianjin, China · Independent Contributor, CA, USA · School of Computer Science and Engineering, Beihang University, Beijing, China

Efficient Language Model Reasoning CoT Compression Continuous Latent Reasoning LLM Interpretability

Open paper AskPDF arXiv

Explicit Chain-of-Thought improves the reasoning performance of large language models but often incurs high inference cost due to verbose token-level traces. While recent approaches reduce this overhead via concise prompting or step pruning, they largely truncate what the model says rather than internalize what the model thinks. Latent reasoning offers a promising alternative by performing computation in the hidden space, yet prior methods face two critical challenges. Many existing approaches rely on rigid point-to-point alignment, forcing a latent token to approximate the final representation of a reasoning step, which can be insufficient to capture the dense, variable-length semantics of an entire reasoning segment. Furthermore, these methods often suffer from a lack of interpretability: latent states are commonly produced by unconstrained optimization or embedding mixing, yielding vectors that are difficult to decode or audit under the pretrained language head. We propose SPOT, a flexible framework that compresses explicit CoT into compact latent pause tokens without enforcing a fixed response template. At the core of SPOT is Span-level Semantic Alignment, a Sinkhorn optimal-transport objective that softly matches each pause token to the semantics of an entire reasoning segment, overcoming the rigidity of step-end alignment. To further improve interpretability, SPOT introduces a Frozen-Head Decoding Constraint that keeps latent states directly decodable as token distributions under the frozen pretrained LM head, enabling readable keyword interpretations of latent thoughts. Experiments on reasoning benchmarks demonstrate that SPOT improves accuracy by 2.3 points on average while reducing generated tokens by 37.5% and provides faithful semantic interpretations of the latent reasoning process.

Abstract

Efficient Language Model Reasoning

Explore similar work

SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains

Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning