SKIP: a Self-knowledge-guided Step-wise Preference Learning Framework for Concise Reasoning

Qinhong Lin; Yuhao Zhang; Yinglun Feng; Zhongliang Yang; Linna Zhou

cs.AISep 15, 2026

SKIP: a Self-knowledge-guided Step-wise Preference Learning Framework for Concise Reasoning

Authors: Qinhong Lin, Yuhao Zhang, Yinglun Feng, Zhongliang Yang, Linna Zhou

Organizations: School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China

Abstract

While Chain-of-Thought (CoT) reasoning has been proven to be effective, it often leads to overthinking, resulting in computational overhead, inference latency, and even degraded performance in large language models (LLMs). Existing concise reasoning frameworks significantly compromise accuracy while compressing the length of output. In this paper, we propose SKIP, a self-knowledge-guided step-wise preference learning framework. Starting with lightweight fine-tuning to adjust the model's output style, SKIP introduces a carefully designed knowledge probing mechanism to guide model to output an answer at each reasoning step. Based on the correctness of intermediate steps, we construct preference data that guide the model toward more efficient and correct reasoning by leveraging DPO. Experimental results demonstrate that our method effectively improves reasoning compression while mitigating performance degradation after fine-tuning. Besides, SKIP shows strong generalization ability on out-of-distribution datasets. We further conducted ablation studies on the component parameters of our framework.

Figures & tables

Fig. 1: Step-wise probe accuracy analysis across reasoning progress. Bin Index represents various stages of the inference process, relative to the complete inference. In the GSM8K and Math datasets, the reasoning exhibits positive returns, while in MedQA, it shows negative returns.

Fig. 2: S elf- K nowledge-gu I ded step-wise P reference learning (SKIP), a three stage reasoning compression framework. We use a simple yet efficient probe to detect whether the model can answer the question. We then train models with preference data to teach them to learn when to exit at specific points.

model	method	GSM8K			Math			MedQA
model	method	Acc	Length	Eff	Acc	Length	Eff	Acc	Length	Eff
	Default	76.70	217.70	0.35	47.20	503.20	0.09	56.83	482.36	0.12
	Budget	72.30	141.50	0.51	44.60	438.20	0.10	40.35	339.80	0.12
	C3OT	68.80	115.30 †	0.60	39.60	320.50	0.12	54.79	101.80	0.54
	BON-1	77.30	149.30	0.52	43.60	392.00	0.11	56.83	99.14	0.57
	SKIP-1	77.70	120.10	0.65 †	45.30 †	376.30	0.12	58.08	97.47	0.60

TABLE I: Results of different reasoning compression methods. The bold text represents the best result for the corresponding metric in the block, and the text with a † represents the second best result for the corresponding metric in the block.

Train data	Method	BBH(DU)			StrategyQA
Train data	Method	Acc	Len	Eff	Acc	Len	Eff
	Default	76.8	257.5	0.298	69	303.8	0.227
GSM8K	BON1	66.9	125.2	0.534	51.3	156.5	0.328
	SKIP-1	72.2	141.7	0.509	63.1	124.8	0.505
	BON8	62.4	88.3	0.706	56.3	142.3	0.395
	SKIP-8	67.0	85.1	0.787	64.1	127.9	0.501

TABLE II: OOD experiment results on Llama-3.1-8B. The bold scores denote the best performance.

Method	DPO-type	GSM8K		MATH
Method	DPO-type	Acc	Len	Acc	Len
SKIP-1	DPO	82.18	145	47.1	412
SKIP-1	Masked-DPO	82.31	142	47.8	401
SKIP-8	DPO	83.96	119	45.8	367.7
SKIP-8	Masked-DPO	84.35	119	46.68	366.9

TABLE III: The difference between DPO and masked-DPO on Llama-3.1-8B. Masked-DPO could achieve better results.

Fig. 3: a.We compared different proportions of knowledge preference data and length preference data to analyze their impact on model performance. b.We analyze the impact of training data size on model performance.

Model	Dataset	Consistency
3B	GSM8K	0.91
	Math	0.89
	StrategyQA	0.87
8B	GSM8K	0.94
	Math	0.87
	StrategyQA	0.85

TABLE IV: The match ratio between EM score and LLM-as-a-judge on the SKIP8 experiment setting. High consistency supports the accuracy of the EM results.

Method	GSM8K			Math
Method	Acc	Len	Eff	Acc	Len	Eff
BON1	81.6	154.0	0.530	46.3	418.5	0.111
SKIP1-V1	82.9	115.2	0.720	47.8	398.2	0.120
SKIP1-V2	82.1	101.7	0.807	43.7	380.6	0.115
BON8	83.4	127.1	0.656	45.1	374.3	0.120
SKIP8-V1	84.4	119.0	0.709	46.7	367.9	0.127

TABLE V: V1 represents the length preference data at

step_{i}

constructed under the condition that ”there is an incorrect answer at

step_{t<i}

”, while V2 represents data that does not need to satisfy this condition. The bold text represents the best result for the corresponding metric in the block.

dataset	BON-8		SKIP-8
dataset	GEN	TRAIN	GEN-SFT/DPO	TRAIN-SFT/DPO
GSM8K	7473*8	7473	3737*8/3736	3737/3169
Math	3750*8	3750	3750*8/3750	3750/2083
MedQA	3000*8	3000	3000*8/3000	3000/1136

TABLE VI: Comparison of inference and training data volumes between BON-8 and SKIP-8 on the Llama3.1-8B-instruct model.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 373 papers

Think Less, Know More: State-Aware Reasoning Compression with Knowledge Guidance for Efficient Reasoning
Apr 10, 2026Yi Sui, Chaozhuo Li, Dawei SongCoT Compression Efficient Language Model Reasoning
Beijing Institute of Technology, China · Beijing University of Posts and Telecommunications, China · The Open University, UK
OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping
Jul 13, 2026Mohammed Ehab, Aymane El Gadarri, Vivek F. Farias +2Efficient Language Model Reasoning Sequential Decision Making
MIT · Columbia Business School
HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps
Jun 2, 2026Xin Liu, Runsong Zhao, Xinyu Liu +8Memory-Augmented Language Models Efficient Language Model Reasoning
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · China Unicom Cloud-Link, China · NiuTrans Research, Shenyang, China

Apr 10, 2026J/K move · Enter open · S save

Think Less, Know More: State-Aware Reasoning Compression with Knowledge Guidance for Efficient Reasoning

Yi Sui, Chaozhuo Li, Dawei Song

Beijing Institute of Technology, China · Beijing University of Posts and Telecommunications, China · The Open University, UK

CoT Compression Efficient Language Model Reasoning Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) achieve strong performance on complex tasks by leveraging long Chain-of-Thought (CoT), but often suffer from overthinking, leading to excessive reasoning steps and high inference latency. Existing CoT compression methods struggle to balance accuracy and efficiency, and lack fine-grained, step-level adaptation to redundancy and reasoning bias. Therefore, we propose State-Aware Reasoning Compression with Knowledge Guidance (STACK), a framework that performs step-wise CoT compression by explicitly modeling stage-specific redundancy sources and integrating with a retrieval-augmented guidance. STACK constructs online long-short contrastive samples and dynamically switches between knowledge-guided compression for uncertain or biased reasoning state and self-prompted compression for overly long but confident state, complemented by an answer-convergence-based early stopping mechanism to suppress redundant verification. We further propose a reward-difference-driven training strategy by combining Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO), enabling models to learn state-conditioned compression strategies. Experiments on three mathematical reasoning benchmarks show that STACK achieves a superior accuracy-efficiency balance, reducing average response length by 59.9% while improving accuracy by 4.8 points over existing methods.

Abstract

Efficient Language Model Reasoning

Explore similar work

Think Less, Know More: State-Aware Reasoning Compression with Knowledge Guidance for Efficient Reasoning

OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping

HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps