Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning

Zhuoyuan Hao; Zhuo Li; Wu Li; Fangming Liu; Min Zhang; Jing Li

cs.CLFeb 6, 2026

Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning

Authors: Zhuoyuan Hao, Zhuo Li, Wu Li, Fangming Liu, Min Zhang, Jing Li

Organizations: Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Huazhong University of Science and Technology, China

Abstract

Recent work improves reasoning in large reasoning models (LRMs) by test-time scaling through self-consistency and parallel thinking, adding generic "thinking tokens", or presenting the question twice, as in re-reading (RE2). However, these approaches either inject task-agnostic tokens or mandate heuristics that do not explain, and often ignore, the spontaneous repetition that many LRMs exhibit at the head of their long chain-of-thought (CoT). In this work, we analyze and harness the model's tendency to restate the question, dubbed Echo of Prompt (EOP), as a front-loaded, compute-shaping mechanism, which can be viewed as self-generated prompt repetition inside the reasoning trace rather than in the input. Specifically, we formalize its probabilistic cost by casting echo removal as rejection-based conditioning, and define the Echo Likelihood Gap $Δ\mathcal{L}$ as a computable proxy linking early repetition to likelihood gains and downstream accuracy. We find that (1) in a causal intervention, reinserting an echo into failed GSM8K traces raises exact match by 7.9-10.4 points for two reasoning models but not for a non-reasoning base model; and (2) in length- and suffix-controlled likelihood analyses and layer-wise attention studies conducted to show that EOP's benefits go beyond verbosity, EOP increases answer to answer-prefix attention in middle layers, consistent with an attention refocusing mechanism. Building on these findings, we develop Echo-Distilled SFT (ED-SFT) to instill an "echo-then-reason" pattern through supervised finetuning, and training-free Echoic Prompting (EP) to re-ground the model mid-trace. We evaluate on GSM8K, MathQA, Hendrycks-MATH, AIME24, and MATH-500 under identical decoding settings and budgets, and find consistent gains over baselines, including thinking-token insertion. Our code is publicly released at https://github.com/hhh2210/echoes-as-anchors.

Figures & tables

Panel of Figure 1 — Figure 1: An illustration of the Echo of Prompt (EOP). Left: An example of a model’s thinking process starting with an echo of the user’s query. Right: The frequency of EOP across several open-source models on the GSM8K dataset, as measured by our trained MLP echo probe (§ 3.1 ).

	Echo Likelihood Gap (per-token)				Extended Echo Metrics
Group	N	Mean $\overline{\Delta\mathcal{L}}$	Std $\sigma(\Delta\mathcal{L})$	Neg. ratio (%)	$\overline{\Delta\mathcal{L}}$ (per token)	$\Delta/\#\text{removed}$	Suffix-only gap	Avg. removed tokens
Correct	819	2.5231	0.7786	0.12	2.4614	0.01103	1.1449	219.7
Wrong	500	2.4421	0.7657	0.00	2.3666	0.01085	1.2938	218.7

Table 1: Echo metrics on GSM8K. Averages over samples in each group, where N denotes the number of samples. The Suffix-only Likelihood Gap measures the per-token log-likelihood difference on the reasoning suffix when conditioned with versus without the echo prefix.

Predictor	Coefficient ( $\beta$ )	Std. Error	P-value
Intercept ( $\beta_{0}$ )	0.12	0.15	0.423
Echo Likelihood Gap ( $\beta_{1}$ )	0.24	0.11	0.022
Echo Length ( $\beta_{2}$ )	0.001	0.0005	0.089

Table 2: Logistic regression results predicting correctness on GSM8K.

Group	N	Accuracy (%)	Last-layer Ans $\to$ Ans-prefix Attn
EOP-present	985	63.8	–
EOP-absent	334	57.2	–
Conditioned on Outcome:
Correct, EOP-present	628	–	0.1374
Correct, EOP-absent	191	–	0.1328
Wrong, EOP-present	357	–	0.1046

Table 3: Comparison of EOP-present and EOP-absent traces on GSM8K (DeepSeek-R1-Distill-Llama-8B). EOP presence is associated with higher accuracy and stronger attention refocusing.

Figure 2: Echo metrics on GSM8K. Left: High-resolution histogram of removed echo-prefix lengths (10-token bins) for correct and wrong traces; most mass lies between roughly 200 and 240 tokens. Right: Echo Likelihood Gap

\Delta\mathcal{L}

(per-token) stratified by removed-prefix length bin; the gap remains positive across all bins.

Metric	Correct	Wrong	Diff (C $-$ W)
Global Statistics
Last-layer: answer $\to$ question	5.77%	5.54%	+0.23%
Last-layer: answer $\to$ answer-prefix	13.69%	10.41%	+3.28%
All-layers mean: answer $\to$ question	8.45%	8.00%	+0.45%
All-layers mean: answer $\to$ answer-prefix	10.64%	8.49%	+2.15%
Peak Effect Layers (7-18)

Table 4: Average attention weights (%) on GSM8K (DeepSeek-R1-Distill-Llama-8B) with probe-estimated prefix length. Global statistics and layer-specific analysis showing attention from answer tokens to question and answer prefix.

Figure 3: Layer-wise attention weight distribution on GSM8K (DeepSeek-R1-Distill-Llama-8B) for Left: answer

\to

answer-prefix and Right: answer

\to

question. The blue lines represent correct reasoning traces while orange lines represent incorrect ones. The attention refocusing effect is most pronounced in layers 7-18 for answer

\to

answer-prefix, with correct traces maintaining consistently higher attention weights.

Layer Group (layers)	AUC $\uparrow$ (Ans $\to$ Pref)	$d$ $\uparrow$ (Ans $\to$ Pref)	AUC (Ans $\to$ Q)	$d$ (Ans $\to$ Q)
Early (0–6)	0.716	0.820	0.628	0.482
Mid (7–18)	0.719	0.832	0.585	0.303
Late (19–31)	0.723	0.828	0.563	0.184

Table 5: Layer-wise discriminability (Correct vs. Wrong) aggregated into layer groups . Metrics are computed on answer

\to

answer-prefix and answer

\to

question attention. The mid-layer group shows the strongest effect size (

d

) for answer

\to

answer-prefix.

Figure 4: Robustness of the answer

\to

answer-prefix gap. Left: raw attention difference (Correct

-

Wrong) and normalized effect size (Cohen’s

d

) across layers; the two track each other, and

d

exceeds 0.75 throughout layers 7–18. Right: attention to a fixed-length prefix of the first

K

answer tokens, averaged over all layers; the Correct

-

Wrong gap stays positive for every

K

Panel of Figure 5 — Figure 5: Fine-grained attention evidence on GSM8K (DeepSeek-R1-Distill-Llama-8B). Top: token-wise attention from the first 32 answer tokens to the answer-prefix (left) and the question (right), with standard-error bands; Correct traces (blue) attend more to the prefix, Wrong traces (orange) more to the question. Bottom: word-level attention from a correct trace to its echo prefix on the “Janet’s ducks” problem (layers 7–18; darker is higher), concentrated on the key quantities.

Panel of Figure 6 — Figure 6: Information flow visualization for generated answer tokens across three different examples. The graphs show the primary attention routes through the model layers. In correct traces, information flows significantly through the Echo of Prompt (EOP) tokens in the middle layers, acting as a bridge between the question and the answer.

Model	Echo-free EM (%)	Echo reinsertion EM (%)
DeepSeek-R1-Distill-Llama-8B	15.85	26.22
Qwen3-8B	21.34	29.27
Qwen3-8B-Base (no CoT)	10.56	10.56

Table 6: Echo reinsertion ablation on the wrong-subset GSM8K traces for several models.

Model	GSM-8K		MathQA	Hendrycks-MATH
	Strict EM	Flex EM
Qwen3-8B-Base
Base	79.4	80.5	31.0	0.76
+ ED-SFT	94.2 (+3.4)	94.2 (+3.4)	58.8 (+11.8)	10.0 (+8.2)
+ normal-SFT	90.8	90.8	47.0	1.8
Qwen3-8B (Instruct Version)

Table 7: Supervised fine-tuning with distilled CoT data improves mathematical reasoning. -ED-SFT denotes models fine-tuned on our EOP distilled dataset. -normal-SFT refers to the normal CoT distilled dataset.Base is the pretrained model; the unmarked variant is instruction-tuned. Best scores in each column are in bold .

Panel of Figure 7 — Figure 7: Echo-then-reason on a Hendrycks-MATH counting problem. Top: the question. Left (green): Qwen3-8B-ED-SFT restates the question first and answers correctly. Right (red): Qwen3-8B-Base jumps straight to a formula without echoing and answers incorrectly.

Figure 8: Echoic Prompting (EP) vs. TTTS on AIME24 (left) and MATH-500 (right).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Accuracy	Precision	Recall	F1-Score	AUROC
Value	0.912	0.921	0.908	0.914	0.963

Appendix

Table 8: MLP probe performance on the held-out test set. The probe reliably identifies echo prefixes, justifying its use in our framework.

Benchmark	Size / Format	Domain / Source	Description (from original papers)
Hendrycks-MATH	12,500 problems; open-ended; step-by-step solutions	Competition mathematics (algebra, geometry, number theory, probability, etc. )	“A new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations.” The evaluation follows the hendrycks_math group in the LM Evaluation Harness, which decomposes MATH into diverse subsets ( _algebra , _counting_and_prob , _geometry , _intermediate_algebra , _num_theory , _prealgebra , _precalc ), covering a wide range of mathematical skills beyond our training distribution.
GSM8K	8.5K problems; free-form answers; word problems	Grade-school math word problems	“A dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution.”
MathQA	37K problems; multiple choice	Multiple math domains; derived from AQuA operation programs	“A large-scale dataset of 37k English multiple-choice math word problems covering multiple math domain categories by modeling operation programs corresponding to word problems in the AQuA dataset.”

Appendix

Table 9: Benchmarks used for our evaluation.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 931 papers

CALM: Consistency-Aware Logit Masking in Decoding Space for Mitigating Overthinking of Large Reasoning Models
Oct 19, 2026Shengjing Liu, Feiyang Li, Yuxuan Yang +2Overthinking in Language Models Efficient Language Model Reasoning
College of Computer Science and Software Engineering, Shenzhen University
Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs
Sep 1, 2026Zhaoliang Chen, Jie FuEfficient Language Model Reasoning Continuous Latent Reasoning
Emory University · IQuest Research
OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping
Jul 13, 2026Mohammed Ehab, Aymane El Gadarri, Vivek F. Farias +2Efficient Language Model Reasoning Sequential Decision Making
MIT · Columbia Business School

Oct 19, 2026J/K move · Enter open · S save

CALM: Consistency-Aware Logit Masking in Decoding Space for Mitigating Overthinking of Large Reasoning Models

Shengjing Liu, Feiyang Li, Yuxuan Yang, Yile Wang+1

College of Computer Science and Software Engineering, Shenzhen University

Overthinking in Language Models Efficient Language Model Reasoning Large Reasoning Models Language Model Decoding Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Large reasoning models (LRMs) have achieved strong performance on challenging mathematical and scientific tasks through long chain-of-thought reasoning, but they often suffer from overthinking with unnecessary reasoning steps. Existing truncation-based methods typically detect consecutive trial-answer agreement and force early exit, yet such local stopping signals do not necessarily indicate final-answer stability and can disrupt useful subsequent reflection or verification, potentially leading to performance degradation. We propose consistency-aware logit masking (CALM), a training-free decoding intervention for mitigating overthinking in LRMs. CALM first monitors trial answers at step boundaries and then applies logit masking to action transition point (ATP) tokens such as "Wait", "Alternatively", and "Hmm", suppressing redundant transitional reasoning while allowing the model to continue its thought process more naturally. Experiments across four advanced LRMs and five benchmarks show that CALM reduces token consumption by 18.6%-25.0% compared with full CoT reasoning while maintaining performance, and can even improve accuracy by 0.07%-1.48% by reducing redundant reasoning. LLM-as-a-Judge results also demonstrate that CALM provides more coherent and concise results.

Abstract

LLM Reasoning

Explore similar work

CALM: Consistency-Aware Logit Masking in Decoding Space for Mitigating Overthinking of Large Reasoning Models

Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs

OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping