Not All Queries Need Deep Thought: CoFiCot for Adaptive Coarse-to-fine Stateful Refinement

Dongxu Zhang; Hongqiang Lin; Yiding Sun; Pengyu Wang; Qirui Wang; Ning Yang; Jihua Zhu

cs.CLMar 9, 2026

Not All Queries Need Deep Thought: CoFiCot for Adaptive Coarse-to-fine Stateful Refinement

Authors: Dongxu Zhang, Hongqiang Lin, Yiding Sun, Pengyu Wang, Qirui Wang, Ning Yang, Jihua Zhu

Organizations: Xi’an Jiaotong University · Zhejiang University · The Chinese University of Hong Kong (Shenzhen) · Institute of Automation, CASIA

Abstract

Scaling test-time computation enhances LLM reasoning ability but faces a uniform computation paradox. Allocating identical resources leads to over-correction on simple tasks and insufficient refinement on complex ones. To address this, we propose CoFiCot, a coarse-to-fine adaptive framework that dynamically tailors inference strategies to problem difficulty. Specifically, we implement a multi-metric classifier that triages queries by synthesizing semantic entropy, consensus reliability, and predicted reasoning depth . This enables a differentiated refinement stage that applies efficient aggregation for simple queries while routing complex ones to a context-aware correction loop . We formalize correction as a stateful sequential propagation process , where each repair is strictly conditioned on the verified history of prior rectifications. By integrating Process Reward Models (PRMs) within this state-dependent trajectory, CoFiCot effectively bridges the gap between granular error localization and global logical coherence, preventing the context fragmentation typical of stateless refinement methods.

Figures & tables

Figure 1: Conceptual illustration of the fundamental paradox of uniform computation in LLM reasoning. (Top) For an Easy Question, the model correctly computes the answer in its initial pass but is forced into unnecessary iterations. This overthinking leads to over correction, where the correct answer is corrupted into a final incorrect one. (Bottom) For a Medium&Hard Question, the same fixed computational budget is insufficient. The reasoning process is prematurely terminated before all logical steps are completed, resulting in an insufficient refinement failure.

Figure 2: The complete workflow of the CoFiCot framework. The process begins in Stage 0, where the base LLM generates an initial ensemble of $k$ reasoning traces. This set is passed to Stage 1, which performs a parallel, multi metric analysis to assess difficulty. Easy problems are resolved with simple aggregation. Medium and Hard problems are channeled into Stage 2. This stage initiates an iterative loop. A PRM scores each step, identifying a flawed step, which is then fed into the Correction Process. Critically, this correction is context aware, conditioning on the history of previous, correct steps to generate a new solution. The refined solution is then selected via an ORM, and the process repeats until termination criteria are met.

Method	MMLU	MATH	SVAMP	GSM8K	SAT	Avg.
Llama3-8B-Instruct
Zero-shot CoT [ Wei et al., 2022 ]	50.4	24.2	72.4	80.1	58.2	57.1
Self-Refine [ Madaan et al., 2023 ]	49.6	24.6	72.0	79.0	57.7	56.6
Best-of- $k$ ( $k$ = 120) [ Lightman et al., 2023 ]	62.6	41.4	88.7	90.1	72.4	71.0
$k$ -way SC ( $k$ = 120) [ Wang et al., 2022 ]	63.0	40.6	89.8	90.3	70.5	70.8
Self-Refine + $k$ -way SC [ Chen et al., 2024a ]	62.1	40.4	88.6	90.1	68.2	69.9

Table 1: Performance comparison of our proposed CoFiCot method against baseline methods on the Llama3-8B-Instruct and GPT-3.5-Turbo models. Results are reported across five reasoning benchmarks. Best performance is highlighted in bold. We report accuracy(%).

Figure 3: Accuracy vs. Effective Sample Size (

k

) on the MATH dataset.

Method	ARC	Date
Zero-shot [ Wei et al., 2022 ]	66.5	52.5
40-way SC [ Wang et al., 2022 ]	85.5	72.5
120-way SC [ Chen et al., 2024a ]	86.0	72.5
CoFiCot(ours)	88.2	80.8

Table 2: Performance comparison on commonsense reasoning and logical reasoning benchmarks. We also report accuracy(%).

Method	MMLU	MATH	SVAMP	GSM8K	SAT	Avg.
Qwen2.5-Math-7B [ Yang et al., 2024 ]	73.9	78.8	91.8	94.9	92.3	86.3
$k$ -way SC ( $k$ = 40) [ Wang et al., 2022 ]	81.3	87.0	95.5	97.2	97.3	91.7
$k$ -way SC ( $k$ = 120) [ Chen et al., 2024a ]	82.0	86.8	95.4	97.3	97.3	91.8
Best-of- $k$ ( $k$ = 120) [ Lightman et al., 2023 ]	82.6	86.0	93.4	96.9	95.2	90.8
CoFiCot(ours)	84.9	91.7	95.8	97.2	97.6	93.4

Table 3: Evaluating CoFiCot’s scalability on the powerful Qwen2.5-Math-7B model. We report accuracy(%).

PRM	ORM	Acc.
Math-Shepherd-7B	InternLM-7B	48.3
Qwen-Math 7B	InternLM-7B	54.9
Math-Shepherd-7B	Llama-3.1-8B	51.2

Table 4: Analysis of CoFiCot’s sensitivity to the quality of its component reward models. Accuracy (Acc. %) is shown for different PRM and ORM combinations.

Figure 4: Token count and accuracy comparison across different datasets. The reported token count for CoFiCot includes the overhead from the initial sampling, Stage 1 classification tokens, and all PRM/ORM verification steps in Stage 2. While scaling Self-Consistency from

k=40

k=120

introduces substantial token overhead, our CoFiCot achieves significantly higher accuracy while being more token efficient than the 120-way SC baseline.

Methods	GSM8K	MATH
Methods	Accuracy	Accuracy
CoFiCot	91.8	47.9
w/o Coarse Stage	88.5 (3.3↓)	45.1 (2.8↓)
w/o Fine Stage	89.1 (2.7↓)	41.2 (6.7↓)

Table 5: Ablation study of the core Coarse and Fine stages of our CoFiCot framework on MATH and GSM8K. We report accuracy(%).

Module			Datasets
Cofi.	Relia.	Comp.	GSM8K	MATH	MMLU	SAT
$\checkmark$	-	-	86.8	43.4	63.9	71.2
-	-	$\checkmark$	87.2	44.6	64.1	70.9
-	$\checkmark$	$\checkmark$	89.7	45.9	67.0	73.4
$\checkmark$	-	$\checkmark$	89.3	46.2	67.4	74.1
$\checkmark$	$\checkmark$	$\checkmark$	91.8	47.9	68.8	75.0

Table 6: Ablation experiments of the CoFiCot. The results are measured in %. We also assess the impact of including (

\checkmark

) or excluding (-) our Coarse stage evaluation metrics, Confidence Assessment(Cofi.) module, Reliability Assessment(Relia.) module and Complexity Assessment(Comp.) module.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 7 — Table 7: The prompt used for estimating $N_{\text{steps}}$ .

Criterion	GSM8K	MATH
Pessimistic	91.4	47.2
Optimistic	90.8	46.5
Democratic	90.3	45.8
CoFiCot(ours)	91.8	47.9

Appendix

Table 8: Ablation study on the criterion for detecting problem difficulty. We compare our method against three alternative strategies: Pessimistic, Optimistic, and Democratic. We report accuracy(%).

Modules	MMLU	MATH
ORM-Only	67.8	47.1
PRM-Only	67.3	46.5
Both	68.8	47.9

Appendix

Table 9: Ablation study on the final answer selection, using ORM-only, PRM-only or both. We also report accuracy(%).

LLM Self-Refinement

Large language model self-refinement studies how models can improve outputs through repeated critique, verification, revision, and feedback.

See all 114 papers

Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
Aug 6, 2026Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer +4Numerical Reasoning in Language Models LLM Self-Correction
University of Oklahoma · Stanford University · Universitat Pompeu Fabra +1
Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs
May 31, 2026Yubo Gao, Haotian Wu, Hong Chen +8LLM Inference Efficiency CoT Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Nanyang Technological University +1
Graph-Based Chain-of-Thought Pruning for Reducing Redundant Reflections in Reasoning LLMs
Apr 7, 2026Hongyuan Yuan, Xinran He, Run Shao +6Efficient Language Model Reasoning Overthinking in Language Models
School of Geosciences and Info-Physics, Central South University, Changsha, China · Baidu Inc., Beijing, China

Aug 6, 2026J/K move · Enter open · S save

Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning

Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer, Lena Trigg+3

University of Oklahoma · Stanford University · Universitat Pompeu Fabra+1

Numerical Reasoning in Language Models LLM Self-Correction LLM Self-Refinement Test-Time Scaling LLM Reasoning

Open paper AskPDF arXiv

Test-time scaling improves LLM reasoning by using additional inference compute, but wider sampling alone can suffer from diminishing returns: new rollouts often repeat existing answer patterns instead of adding useful reasoning diversity. Verifier-based selection offers an alternative, but its performance depends on the calibration of an external reward model. We propose a verifier-free breadth--depth refinement framework that uses test-time compute to both explore and improve candidate solutions. The method samples multiple independent reasoning rollouts, refines each rollout through iterative self-critique and self-correction, and aggregates the refined answers by majority voting. Breadth preserves diverse initial attempts, while depth repairs local reasoning errors before aggregation. Across AIME24, AIME25, AMC, OlympiadBench, and MATH500, our method consistently improves over greedy decoding, majority voting, verifier-based best-of-

N

, beam search, and lookahead decoding across multiple open-weight models. For instance, with Qwen2.5-1.5B, accuracy increases from the strongest verifier-based baseline to

58.0\%

on MATH500, and from

25.0\%

32.5\%

on AMC. These results show that test-time compute can be more effective when used to refine sampled trajectories rather than only to sample more candidates or rely on verifier-guided selection.

Abstract

LLM Self-Refinement

Explore similar work

Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning

Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs

Graph-Based Chain-of-Thought Pruning for Reducing Redundant Reflections in Reasoning LLMs