Mitigating Query Shortcuts in Chain-of-Thought with Context-Aware Decoding

Shin'ya Yamaguchi

cs.CLOct 21, 2026

Mitigating Query Shortcuts in Chain-of-Thought with Context-Aware Decoding

Authors: Shin'ya Yamaguchi

Organizations: NTT

Abstract

While Chain-of-Thought (CoT) prompting improves the reasoning performance of large language models (LLMs), recent studies have shown that final answers do not always depend on the generated rationales. In particular, LLMs may rely on query-specific shortcuts rather than the information provided by their reasoning steps. In this paper, we investigate whether context-aware decoding (CAD), an existing contrastive decoding method, can mitigate such shortcut behavior when applied to CoT reasoning. We study this application as Rationale Contrastive Decoding (RCD), which contrasts the next-token distributions conditioned on the query with and without its rationale, amplifying tokens whose likelihood increases due to the rationale. We provide a theoretical interpretation of this decoding rule via a KL-regularized reward-maximization objective. Experiments across five reasoning benchmarks demonstrate that RCD improves answer accuracy over standard CoT. Furthermore, intervention analyses using ground-truth and random rationales reveal that RCD increases the behavioral dependence of final answers on the rationale content. Our findings suggest that contrastive decoding offers a simple, training-free approach to mitigating query shortcuts.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of Rationale Contrastive Decoding (RCD), which applies context-aware decoding to CoT answer generation. Instead of standard decoding, RCD contrasts the rationale-conditioned pass and the query-only pass to explicitly strengthen the dependency on the rationales, without any parameter updates.

	GSM8K
	Self ( $\uparrow$ )	GT ( $\uparrow$ )	Random ( $\downarrow$ )
	Qwen3-8B
Direct	50.14 $\pm$ .17	$-$	$-$
CoT	89.16 $\pm$ .13	94.93 $\pm$ .04	43.35 $\pm$ .03
Logit Addition	81.26 $\pm$ .40	83.41 $\pm$ .68	50.82 $\pm$ .72
RCD	89.91 $\pm$ .28	95.82 $\pm$ .01	27.79 $\pm$ .03

Table 1: Intervention Analysis with Ground-Truth (GT) and Irrelevant (Random) Rationales.

	GSM8K	MATH	CommonsenseQA	AQuA-RAT	MMLU-Pro
	Qwen3-8B
Direct	50.14 $\pm$ .17	1.73 $\pm$ .23	71.27 $\pm$ .81	58.88 $\pm$ .74	35.29 $\pm$ .48
CoT	89.16 $\pm$ .13	17.20 $\pm$ .35	78.29 $\pm$ .53	68.50 $\pm$ .68	50.24 $\pm$ .52
Logit Addition	81.26 $\pm$ .40	15.59 $\pm$ .66	77.19 $\pm$ .56	63.38 $\pm$ .53	48.62 $\pm$ .65
RCD	89.91 $\pm$ .28	27.60 $\pm$ .51	80.13 $\pm$ .47	70.41 $\pm$ .40	52.28 $\pm$ .41

Table 2: Performance on Reasoning Benchmarks.

Figure 2: Qualitative Evaluation with Qwen3-8B. Top: Given a self-generated rationale supporting answer (A), standard CoT predicts (E), whereas RCD follows the rationale and predicts (A). Bottom: Given an irrelevant rationale randomly sampled from another question, standard CoT produces the correct answer, whereas RCD indicates that the rationale is unrelated to the query. These examples illustrate RCD’s stronger sensitivity to rationale content.

Rationale	CoT Acc.	RCD Acc.	Wrong $\to$ Correct	Correct $\to$ Wrong
	Qwen3-8B
Correct ( $N$ =1,201)	97.25	98.08	10/33 (30.30%)	0/1,168 (0.00%)
Incorrect ( $N$ =118)	6.78	6.78	0/110 (0.00%)	0/8 (0.00%)

Table 3: Prediction Transitions from CoT to RCD

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Artifact	Type	License / Terms
GSM8K	Dataset	MIT License
MATH	Dataset	MIT License
CommonsenseQA	Dataset	MIT License
AQuA-RAT	Dataset	MIT License
MMLU-Pro	Dataset	Apache License 2.0
Qwen3	Model	Apache License 2.0

Appendix

Table 4: Licenses or terms of use for the public artifacts used in this work, based on their public dataset or model cards.

Figure 3: Effects of

\alpha

on GSM8K validation accuracy (%) using Qwen3-8B. We evaluate self-generated (Self), ground-truth (GT), and randomly shuffled (Random) rationales on 1,000 validation examples. Increasing (

\alpha

) generally reduces accuracy with random rationales, while excessively large values degrade accuracy with self-generated rationales, illustrating the trade-off between rationale reliance and answer accuracy.

	GSM8K
Model	Direct	CoT	RCD
Qwen3-0.6B	23.49 $\pm$ .11	47.44 $\pm$ .18	47.96 $\pm$ .22
Qwen3-8B	50.14 $\pm$ .17	89.16 $\pm$ .13	89.91 $\pm$ .28
Qwen3-14B	38.38 $\pm$ .43	89.61 $\pm$ .22	90.31 $\pm$ .20
Qwen3-32B	48.23 $\pm$ .37	91.31 $\pm$ .33	93.27 $\pm$ .28
Gemma3-4B	8.59 $\pm$ .03	80.69 $\pm$ .13	81.93 $\pm$ .03

Appendix

Table 5: GSM8K test accuracy (%) across different model families and sizes, including Qwen3, Gemma3, and Llama-3.1. We compare direct answering (Direct), standard CoT, and RCD with (

\alpha=1

). CoT and RCD share the same generated rationales. RCD consistently improves accuracy over standard CoT across all evaluated models, demonstrating its applicability across model architectures and scales.

	Latency (s/query)	Peak VRAM (GB)
	Qwen3-8B
Direct	1.506	15.33
CoT	2.823	15.64
RCD	2.864	15.86

Appendix

Table 6: Computational Analysis (GSM8K)

Method	$k$	Accuracy (%)	Latency (s/query)
CoT	1	89.16	2.82
RCD	1	89.91	2.86
SC	3	89.58	8.63
SC + RCD	3	90.79	8.65

Appendix

Table 7: Comparison with Self-Consistency (SC) on GSM8K using Qwen3-8B.

k

denotes the number of sampled rationales. Latency is measured end-to-end per query.

CoT Reasoning

Chain-of-thought (CoT) reasoning studies how language models use intermediate steps, visible or latent, to solve multi-step tasks.

See all 228 papers

DoubtLess: Training-Free Contrastive Decoding for Efficient Reasoning
Oct 13, 2026Minghao Luo, Liang ChenEfficient Language Model Reasoning CoT Compression
The Chinese University of Hong Kong · EPFL
CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning
May 19, 2026Dachuan Shi, Hanlin Zhu, Xiangchi Yuan +4Agentic Reasoning Efficient Language Model Inference
Georgia Tech · UC Berkeley · Stanford University +1
CASE: Causal Alignment and Structural Enforcement for Improving Chain-of-Thought Faithfulness
Jul 21, 2026Ziming Wang, Yinghua Yao, Changwu Huang +2CoT Faithfulness Faithfulness of Language Model Explanations
Department of Computer Science and Engineering of Southern University of Science and Technology, Shenzhen, China · Center for Frontier AI Research, Agency for Science, Technology and Research (A*STAR), Singapore · School of AI and Liberal Arts, Beijing Normal-Hong Kong Baptist University, Zhuhai, China +1

Oct 13, 2026J/K move · Enter open · S save

DoubtLess: Training-Free Contrastive Decoding for Efficient Reasoning

Minghao Luo, Liang Chen

The Chinese University of Hong Kong · EPFL

Efficient Language Model Reasoning CoT Compression Contrastive Decoding

Open paper AskPDF arXiv

Reasoning models achieve strong performance through long chains of thought (CoT), but often spend more tokens than needed to solve a problem. Existing ways to shorten the CoT require additional training for each model, add a fixed steering vector to the model's hidden states, or use decoding heuristics such as stopping early or banning words like "wait", which can reduce accuracy. We propose DoubtLess, a training-free decoding method in which a much smaller reasoning model steers the large one towards brevity. The small model reads the large model's CoT so far twice, once prompted to be brief (the brief thinker) and once prompted to be verbose (the verbose thinker), and the difference between its two logit vectors, scaled by a strength, is added to the large model's logits at every step of the CoT. On seven benchmarks of competition mathematics, graduate-level science and competitive programming, DoubtLess shortens the CoT of Qwen3-32B by 31% with unchanged accuracy, outperforming five inference-time baselines. We also show that DoubtLess reduces thinking tokens per turn by 20% in a self-debugging coding agent with unchanged task performance. It also generalizes to two other model families with unchanged accuracy. When combined with training-based compression, it further shortens the CoT of a model already trained for brevity. Ablations and behavioral analysis show that DoubtLess reduces repeated checking, with most token savings coming after the correct answer first appears in the CoT. Our code is available at https://github.com/leoluolol/DoubtLess.

Abstract

CoT Reasoning

Explore similar work

DoubtLess: Training-Free Contrastive Decoding for Efficient Reasoning

CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

CASE: Causal Alignment and Structural Enforcement for Improving Chain-of-Thought Faithfulness