Thinking in Latents: Adaptive Anchor Refinement for Implicit Reasoning in LLMs

Disha Sheshanarayana; Rajat Subhra Pal; Manjira Sinha; Tirthankar Dasgupta

cs.CLMar 16, 2026

Thinking in Latents: Adaptive Anchor Refinement for Implicit Reasoning in LLMs

Authors: Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha, Tirthankar Dasgupta

Organizations: Manipal University Jaipur · TCS Research

Abstract

Token-level Chain-of-Thought (CoT) prompting has become a standard way to elicit multi-step reasoning in large language models (LLMs), especially for mathematical word problems. However, generating long intermediate traces increases output length and inference cost, and can be inefficient when the model could arrive at the correct answer without extensive verbalization. This has motivated latent-space reasoning approaches that shift computation into hidden representations and only emit a final answer. Yet, many latent reasoning methods depend on a fixed number of latent refinement steps at inference, adding another hyperparameter that must be tuned across models and datasets to balance accuracy and efficiency. We introduce AdaAnchor, a latent reasoning framework that performs silent iterative computation by refining a set of latent anchor vectors attached to the input. AdaAnchor further incorporates an adaptive halting mechanism that monitors anchor stability across iterations and terminates refinement once the anchor dynamics converge, allocating fewer steps to easier instances while reserving additional refinement steps for harder ones under a shared maximum-step budget. Our empirical evaluation across three mathematical word-problem benchmarks shows that AdaAnchor with adaptive halting yields accuracy gains of up to 5% over fixed-step latent refinement while reducing average latent refinement steps by 48-60% under the same maximum-step budget. Compared to standard reasoning baselines, AdaAnchor achieves large reductions in generated tokens (92-93%) by moving computation into silent latent refinement, offering a different accuracy-efficiency trade-off with substantially lower output-token usage.

Figures & tables

Figure 1: Comparison of AdaAnchor with explicit Chain-of-Thought (CoT) reasoning. CoT generates long intermediate reasoning tokens, whereas AdaAnchor performs implicit multi-step computation by refining latent anchor vectors and uses stability-based early stopping before answer-only decoding.

Figure 2: Overview of AdaAnchor. AdaAnchor prepends $m$ learnable latent anchor vectors to the input embedding sequence (left), iteratively refines them via repeated forward passes and anchor-slot updates (middle), and uses a stability-based criterion to halt early before performing answer-only decoding (right).

Dataset	Task Type	Metric
GSM8K ( Cobbe et al., 2021 )	Grade-school mathematical word problems	Accuracy
SVAMP ( Patel et al., 2021 )	Arithmetic word problems	Accuracy
MultiArith ( Roy and Roth, 2015 )	Multi-step arithmetic word problems	Accuracy

Table 1: Overview of datasets used.

		GSM8K			SVAMP			MultiArith
Model	Method	Acc.	Avg Tok	Avg Steps	Acc.	Avg Tok	Avg Steps	Acc.	Avg Tok	Avg Steps
Qwen2.5-1.5B	No CoT	13.0	2.16	–	42.0	2.34	–	22.3	2.41	–
	CoT	20.0	28.27	–	59.3	29.09	–	34.3	30.2	–
	iCoT	12.23	2.36	–	48.5	2.04	–	28.56	1.66	–
	AdaAnchor ( $K{=}8$ )	16.0	2.73	8	50.5	2.12	8	27.6	2.34	8
	AdaAnchor (adaptive)	16.0	2.17	3.23	55.2	2.23	4.12	29.4	2.16	3.82

Table 2: Experimental results on mathematical reasoning benchmarks. We report accuracy (Acc. %), average output tokens (Avg Tok), and average latent refinement steps (Avg Steps) for answer-only evaluation. No CoT denotes direct answer generation without reasoning. CoT generates full chain-of-thought rationales before the answer. iCoT uses implicit chain-of-thought.

Figure 3: Accuracy vs. fixed latent refinement budget $K$ on Qwen2.5-1.5B. Each panel reports performance as a function of $K\in\{1,2,4,8\}$ .

Figure 4: Adaptive halting distribution showing percentage of examples halting at each step (1-8) across datasets on Qwen2.5-1.5B.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 370 papers

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains
May 25, 2026Hui Xie, Jie Liu, Ziyue Qiao +1Efficient Language Model Reasoning LLM Inference Acceleration
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models
Mar 6, 2026Yunlong Chu, Minglai Shao, Yuhang Liu +4Efficient Language Model Reasoning CoT Compression
School of New Media and Communication, Tianjin University, Tianjin, China · Independent Contributor, CA, USA · School of Computer Science and Engineering, Beihang University, Beijing, China
A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM
Sep 7, 2026Xiaoang Xu, Siyuan Liu, Shuo Wang +13Efficient Language Model Reasoning Continuous Latent Reasoning
Beijing University of Posts and Telecommunications · OpenBMB · The Hong Kong Polytechnic University +2

May 25, 2026J/K move · Enter open · S save

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains

Hui Xie, Jie Liu, Ziyue Qiao, Joaquin Vanschore

Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China

Efficient Language Model Reasoning LLM Inference Acceleration CoT Compression LLM Reasoning Latent Reasoning Latent CoT Reasoning

Open paper AskPDF arXiv

Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models (LLMs), but incurs high inference cost due to lengthy autoregressive traces. Existing latent reasoning methods offer a promis- ing alternative, yet they often treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed and thereby degrading reasoning accuracy. In this work, we propose Selective Latent Thinking (SLT), a reasoning paradigm that selectively compresses redundant reasoning spans into latent representations while preserving precision-critical spans as explicit CoT within the same reasoning trajectory. Our primary instantiation uses a lightweight decoder to anticipate a short upcoming reasoning span and applies confidence- based gating to determine the longest span that can be reliably compressed; the resulting selective compression policy is further optimized with trajectory-level reinforcement learning to balance answer correctness against reasoning cost. Ex- periments across four mathematical reasoning benchmarks show that which spans are compressed is critical: with the same compressor, compressing arbitrary spans reduces the GSM-Hard accuracy of Qwen3-4B from 54.7% to 17.3%, whereas keeping numerical tokens explicit retains 51.8%. SLT reduces explicit reasoning length by 58.4% with only a 2.8-point accuracy drop compared to explicit CoT on Llama-3.2-1B. Notably, the principle extends beyond this design: instantiated within existing latent reasoning models via a learned compression gate, it improves accuracy at equal reasoning length (+2.6 points over Latent-SFT) and at shorter length (+5.6 points over Latent-GRPO with 13% fewer tokens).

Abstract

Efficient Language Model Reasoning

Explore similar work

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains

SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models

A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM