GCoT-Decoding: Unlocking Deep Reasoning Paths for Universal Question Answering

Guanran Luo; Wentao Qiu; Zhongquan Jian; Meihong Wang; Qingqiang Wu

cs.CLApr 8, 2026

GCoT-Decoding: Unlocking Deep Reasoning Paths for Universal Question Answering

Authors: Guanran Luo, Wentao Qiu, Zhongquan Jian, Meihong Wang, Qingqiang Wu

Organizations: School of Informatics, Xiamen University

Abstract

Chain-of-Thought reasoning can enhance large language models, but it requires manually designed prompts to guide the model. Recently proposed CoT-decoding enables the model to generate CoT-style reasoning paths without prompts, but it is only applicable to problems with fixed answer sets. To address this limitation, we propose a general decoding strategy GCoT-decoding that extends applicability to a broader range of question-answering tasks. GCoT-decoding employs a two-stage branching method combining Fibonacci sampling and heuristic error backtracking to generate candidate decoding paths. It then splits each path into a reasoning span and an answer span to accurately compute path confidence, and finally aggregates semantically similar paths to identify a consensus answer, replacing traditional majority voting. We conduct extensive experiments on six datasets covering both fixed and free QA tasks. Our method not only maintains strong performance on fixed QA but also achieves significant improvements on free QA, demonstrating its generality.

Figures & tables

	Free QA	Fixed QA
Example	Q : What do Woodrow Wilson, George W. Bush, and James Monroe have in common? k=1 : They all served as presidents of the United States . k=2 : They were all American leaders involved in major wars. k=3 : Each of them occupied the White House as U.S. president .	Q : A factory makes 3 toys per hour. How many toys after 8 hours? k=1 : $3\times 8=\textbf{24}$ (0.93) k=2 : 3 times 8 is 24 (0.91) k=3 : = 24 (0.85)
Answer Space	$\infty$	$\mathbb{N}$
Exact Span Match	$\times$	$\checkmark$
Majority Vote Aggregation	$\times$	$\checkmark$

Table 1: Comparison of CoT-decoding in free-form vs. fixed-format QA tasks.

Figure 1: Overall workflow of GCoT-decoding. A small number of reasoning paths are first generated via Fibonacci seeding and confidence-based backtracking, then scored by length-aware logit gaps, and finally aggregated through greedy semantic clustering over answer strings.

	Spec Ans	GSM8K			MultiArith			Sports understanding
		Mistral - 7B	Gemma - 7B	Llama - 3.1 - 8B	Mistral - 7B	Gemma - 7B	Llama - 3.1 - 8B	Mistral - 7B	Gemma - 7B	Llama - 3.1 - 8B
Greedy	×	10.5	11.6	17.9	16.0	18.7	38.8	49.6	61.2	51.6
Temperature sampling	×	8.4	7.9	13.1	15.2	18.8	36.2	48.9	60.1	52.4 $\dagger$
Top - k sampling	×	5.1	6.2	14.2	13.3	17.3	37.0	50.3	58.0	51.9
Beam search	×	6.7	10.2	17.1	15.5	17.9	38.1	48.2	59.9	50.7
CoT - decoding	$\checkmark$	21.9 $\spadesuit$	25.4 $\spadesuit$	36.3 $\dagger$	40.6 $\spadesuit$	43.8 $\spadesuit$	72.3 $\dagger$	50.6	68.4 $\spadesuit$	51.0

Table 2: Accuracy comparison of decoding strategies on fixed QA tasks; the top - ranked is marked with

\spadesuit

and the second - ranked is marked with

\dagger

. Spec Ans indicates whether the decoding strategy relies on specific answer spans. The top section lists single-path decoding strategies; the bottom section shows multi-path decoding strategies.

	SQuAD v1.1 (contextual)						BARQA (contextual)						Auto categorization (context - free)
	Gemma - 7B		Llama - 3.1 - 8B		Qwen2.5 - 14B		Gemma - 7B		Llama - 3.1 - 8B		Qwen2.5 - 14B		Gemma - 7B		Llama - 3.1 - 8B		Qwen2.5 - 14B
	BLEU	MATCH	BLEU	MATCH	BLEU	MATCH	BLEU	MATCH	BLEU	MATCH	BLEU	MATCH	BLEU	MATCH	BLEU	MATCH	BLEU	MATCH
Greedy	3.3	42.8	8.3	60.6	21.4	67.2	4.7	36.6 $\dagger$	10.8	39.7	10.7 $\dagger$	44.4 $\spadesuit$	5.8	16.8	5.1 $\dagger$	16.0 $\dagger$	8.5	29.0
Temperature sampling	3.1	40.1	7.5	57.2	17.1	64.1	4.5	32.1	7.3	37.4	7.7	42.5	6.0	13.6	4.9	13.3	6.6	27.9
Top - k sampling	2.8	35.2	5.4	51.0	13.1	55.1	2.9	33.3	6.8	37.2	6.4	40.0	4.3	13.7	4.5	11.2	5.6	26.0
Beam Search	3.2	41.9	7.9	59.3	20.0	66.0	4.2	35.4	10.0	38.5	10.1	42.1	5.3	15.0	4.7	15.4	8.1	28.4

Table 3: Performance of different models on free QA tasks; the top - ranked is marked with

\spadesuit

and the second - ranked is marked with

\dagger

. The top section lists single-path decoding strategies; the bottom section shows multi-path decoding strategies.

Figure 2: The results of combining GCoT-decoding with CoT prompting.

Variant	GSM8K (Gemma-7B)	GSM8K (Mistral-7B)	SQuAD v1.1 (Gemma-7B)	SQuAD v1.1 (Llama-3.1-8B)
Fibonacci + greedy (ours)	21.8	18.0	54.6	67.2
top- $k$ sampling ( $k{=}10$ )	7.9	6.2	42.1	50.4
top- $p$ sampling ( $p{=}0.9$ )	8.6	7.0	43.5	51.3
temperature sampling ( $T{=}0.7$ )	9.4	7.8	45.0	52.6

Table 4: Ablation of path-generation strategies under a fixed budget of

K{=}10

paths. All variants share the same backtracking and aggregation modules.

Variant	Backtracking trigger rate (%)	Success rate given trigger (%)	MATCH	BLEU
No-backtracking	–	–	52.7	8.7
Random backtracking	100.0	18.1	52.0	8.6
Late backtracking	100.0	20.4	51.8	8.5
Local-minima backtracking (ours)	28.0	36.5	54.6	9.1

Table 5: Backtracking variants on SQuAD v1.1 dev (Gemma-7B); “Success rate given trigger” is the fraction of triggered cases corrected by backtracking.

Aggregation variant	Extra time per question (sec.)	GSM8K Acc. (Gemma-7B)	SQuAD MATCH (Gemma-7B)
MaxPath (no aggregation)	0.0	15.3	41.9
Greedy clustering (ours)	0.2	21.8	54.6
LLM-based aggregation	8.3	22.1	55.8

Table 6: MaxPath vs. greedy semantic clustering and an LLM-based aggregation module (Gemma-7B). Extra time is measured relative to greedy decoding.

Panel of Figure 3 — Figure 3: The impact of model size and the number of decoding paths $k$ .

	Question: AUSTRO-ITALIAN WAR, JACOBITE REBELLION, and FRANCO-SPANISH WAR are instances of		Question: Profitable home Chelisheva, The House with Lions, and House under the steeple can be classified as
	Ground truth: historical wars		Ground truth: tourist attractions / architecture in Russia
k=1	European diplomatic initiatives. So the answer is: European diplomatic initiatives ( $\Delta$ =0.22)	×	These are notable tourist attractions located across Russia. So the answer is: tourist attractions ( $\Delta$ =0.81)	✓
k=2	diplomatic initiatives. So the answer is: diplomatic initiatives. ( $\Delta$ =0.18)	×	architectural heritage in Russia. So the answer is: architecture in Russia ( $\Delta$ =0.68)	✓
k=3	These events can be categorized under diplomatic initiatives. So the answer is: diplomatic initiatives ( $\Delta$ =0.09)	×	tourist attractions in Russia. Explanation: each of these locations is a notable architectural site known for its historical significance within Russian cities. So the answer is: tourist attractions ( $\Delta$ =0.93)	✓
k=5	They are relevant to international treaty formation. So the answer is: international treaty formation ( $\Delta$ =0.14)	×	They refer to government-owned residential complexes. So the answer is: government-owned residential complexes ( $\Delta$ =0.24)	×
k=8	Historical wars, because each conflict exemplifies armed struggles … So the answer is: historical wars ( $\Delta$ =0.81)	✓	metaphors from Soviet-era literature about class struggle. So the answer is: Soviet-era literature ( $\Delta$ =0.11)	×

Table 7: Decoding outputs with confidence gaps

\Delta_{k,\text{answer}}

for two classification examples.

Figure 4: Illustration of early path backtracking.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Impact of different answer extraction strategies on CoT-decoding performance.

Index	Correct	Incorrect	C. Conf.	I. Conf.
0	–	–	–	–
1	8	92	0.73	0.09
2	2	98	0.68	0.13
3	13	87	0.70	0.10
4	23	77	0.74	0.14
$\cdots$	$\cdots$	$\cdots$	$\cdots$	$\cdots$

Appendix

Table 8: Distribution of correct and incorrect paths and their corresponding confidences for the top 100 GSM8K questions in the case of first-index error.

b=\begin{cases}\min S,&S\neq\varnothing\\ -1,&S=\varnothing\end{cases}

Appendix

Algorithm 1 General decoding path generation with Fibonacci sampling and backtracking

Confidence Computation	Path Generation	Path Aggregation	GSM8K (Acc., Gemma-7B)	GSM8K (Acc., Mistral-7B)	SQuAD v1.1 (MATCH, Gemma-7B)	SQuAD v1.1 (MATCH, Llama-3.1-8B)
–	–	–	21.8	18.0	54.6	67.2
entropy	–	–	18.3 (–3.5)	14.1 (–3.9)	51.4 (–3.2)	63.0 (–4.2)
logits	–	–	19.0 (–2.8)	15.7 (–2.3)	54.5 (–0.1)	66.9 (–0.3)
–	Seq	–	17.5 (–4.3)	13.1 (–4.9)	48.9 (–5.7)	63.6 (–3.6)
–	OneBranch	–	20.8 (–1.0)	16.5 (–1.5)	52.7 (–1.9)	67.0 (–0.2)
–	–	MaxPath	15.3 (–6.5)	12.8 (–5.2)	41.9 (–12.7)	50.8 (–16.4)

Appendix

Table 9: Ablation study of GCoT-Decoding. Top rows show single-factor ablations; bottom rows show selected multi-factor variants. Numbers in parentheses denote drops from the full model.

$\tau$	$\delta$	GSM8K			MultiArith			Sports Underst.
		Mistral-7b	Gemma-7b	Llama-3.1-8b	Mistral-7b	Gemma-7b	Llama-3.1-8b	Mistral-7b	Gemma-7b	Llama-3.1-8b
0.8	0.2	18.0	21.8	41.7	31.3	23.2	74.3	52.0	65.2	58.0
0.7	0.2	16.9	20.5	40.8	30.1	21.9	72.6	49.8	63.7	55.3
0.9	0.2	17.3	21.0	40.9	30.5	22.7	73.2	51.5	64.2	57.0
0.8	0.1	17.2	21.1	41.1	30.7	22.4	73.4	51.7	64.5	57.3
0.8	0.3	17.4	21.4	41.2	30.6	22.6	73.7	51.6	64.7	57.5

Appendix

Table 10: Performance under different thresholds

\tau

and

\delta

on GSM8K, MultiArith, and Sports Understanding tasks.

Figure 6: Prompting examples used in the SQuAD dev-v1.1 task under different few-shot settings. Zero-shot uses no demonstrations, one-shot includes only Example 1, and three-shot includes all three examples.

Figure 7: Prompting examples used in different few-shot settings for the GSM8K task, adapted to arithmetic reasoning.

Category	Method	Gemma-7B	Llama-3.1-8B
Clustering	Greedy Clustering	54.6	67.2
	K-Means++	54.4	66.9
	Agglomerative (Ward)	54.7	67.1
	Spectral Clustering	54.5	67.0
Representative	First-in-Cluster	54.6	67.2
	Cluster Centroid	47.8	60.4

Appendix

Table 11: Accuracy comparison of clustering methods and representative choices on SQuAD v1.1.

Figure 8: Effect of the maximum number of backtracking points per path under a fixed overall path budget.

Method	GSM8K (Acc.)			MultiArith (Acc.)			Sports Understanding (Acc.)
Method	Mistral-7B	Gemma-7B	Llama-3.1-8B	Mistral-7B	Gemma-7B	Llama-3.1-8B	Mistral-7B	Gemma-7B	Llama-3.1-8B
GCoT-decoding + SpanAlign (Last)	10.7	15.4	34.0	16.8	19.7	69.3	48.0	67.2	52.0
GCoT-decoding + SpanAlign (Mean)	10.2	14.9	33.5	16.1	19.0	68.5	47.1	66.3	51.4

Appendix

Table 12: Comparison between using only the last aligned answer span (SpanAlign (Last)) and averaging over all aligned spans (SpanAlign (Mean)).

Template	SQuAD v1.1 MATCH (Gemma-7B)
“So the answer is …”	54.6
“Therefore, the answer is …”	54.5
“Final answer:”	54.3

Appendix

Table 13: Ablation on answer-extraction templates for GCoT-decoding on SQuAD v1.1.

Setting	SQuAD v1.1 BLEU	SQuAD v1.1 MATCH	Auto-cat BLEU	Auto-cat MATCH
GCoT + MiniLM	10.0	67.2	10.6	30.5
GCoT + MPNet-base	9.8	66.7	10.4	30.3
GCoT + E5-small	10.1	67.0	10.5	30.4

Appendix

Table 14: Embedding model ablation for the semantic clustering module in GCoT-decoding.

Question : What position does Von Miller play? Path1(×) : Von Miller plays(0.0877) defensive(0.0921) end position .(0.1980) Path2(✓) : Von plays the outside linebacker position on his team . Path3(×) : Von Miller plays the defensive end role for his team and is known for his pass rushing ability .

Appendix

Table 15: An example of path backtracking. The underlined segments indicate the answers targeted by the decoding paths, while the highlighted portions show the content generated after backtracking. “plays”, “defensive”, and “.” are the three local minima in Path1.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 926 papers

Mitigating Query Shortcuts in Chain-of-Thought with Context-Aware Decoding
Oct 21, 2026Shin'ya YamaguchiCoT Reasoning Faithfulness of Language Model Reasoning
NTT
From Chain-of-Thought to Loops: Non-Autoregressive Latent Reasoning via Looped Transformers
Oct 8, 2026Gerard Grau García, Arnau Padrés Masdemont, Niccolò Grillo +3Efficient Language Model Reasoning Continuous Latent Reasoning
Qualcomm AI Research · MIT · University of Cambridge +1
Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate
Jun 15, 2026Hanyu Lin, Min Cai, Jiawei Wen +1Latent Reasoning LLM Reasoning
1Shenzhen University · University of Alberta

Oct 21, 2026J/K move · Enter open · S save

Mitigating Query Shortcuts in Chain-of-Thought with Context-Aware Decoding

Shin'ya Yamaguchi

NTT

CoT Reasoning Faithfulness of Language Model Reasoning Contrastive Decoding

Open paper AskPDF arXiv

While Chain-of-Thought (CoT) prompting improves the reasoning performance of large language models (LLMs), recent studies have shown that final answers do not always depend on the generated rationales. In particular, LLMs may rely on query-specific shortcuts rather than the information provided by their reasoning steps. In this paper, we investigate whether context-aware decoding (CAD), an existing contrastive decoding method, can mitigate such shortcut behavior when applied to CoT reasoning. We study this application as Rationale Contrastive Decoding (RCD), which contrasts the next-token distributions conditioned on the query with and without its rationale, amplifying tokens whose likelihood increases due to the rationale. We provide a theoretical interpretation of this decoding rule via a KL-regularized reward-maximization objective. Experiments across five reasoning benchmarks demonstrate that RCD improves answer accuracy over standard CoT. Furthermore, intervention analyses using ground-truth and random rationales reveal that RCD increases the behavioral dependence of final answers on the rationale content. Our findings suggest that contrastive decoding offers a simple, training-free approach to mitigating query shortcuts.

Abstract

LLM Reasoning

Explore similar work

Mitigating Query Shortcuts in Chain-of-Thought with Context-Aware Decoding

From Chain-of-Thought to Loops: Non-Autoregressive Latent Reasoning via Looped Transformers

Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate