SCA: Segment-Wise CoT Compression with Answer Alignment

Ye Tian; Hongyu Lin

cs.AIMar 8, 2026

SCA: Segment-Wise CoT Compression with Answer Alignment

Authors: Ye Tian, Hongyu Lin

Abstract

Chain-of-thought (CoT) reasoning improves problem solving, but long think traces increase inference cost. Existing CoT compression methods usually optimize completion-level length. For structured thinking models, however, a completion contains both a think segment and an answer segment, so completion-level compression can save tokens by compressing not only the CoT but also the answer. We call this failure mode answer drift. We propose Segment-wise CoT Compression with Answer Alignment (SCA), an answer-preserving think-compression method. SCA parses completions into functional segments, routes compression rewards only to successful think tokens, and protects answer tokens through length and distribution alignment to a frozen base model. Experiments show that, across datasets from multiple domains, SCA achieves state-of-the-art-level chain-of-thought compression while preserving the base model's performance and answer alignment. Training data and code are included in the supplementary code and data package.

Figures & tables

Figure 1: Example of answer drift and the desired behavior of SCA.

Figure 2: Overview of SCA’s segment-wise training pipeline.

Panel of Figure 3 — Figure 3: Training dynamics for validation think length and answer-length reward during SCA training.

Model	Method	MATH-500	AMC23	MinervaMath	AIME24	AIME25	Avg.
Qwen3-4B	Base	$97.2{\pm}0.3$	$97.5{\pm}2.3$	$69.9{\pm}0.4$	$70.0{\pm}3.1$	$73.3{\pm}3.1$	81.59
Qwen3-4B	DeepCompress	$97.0{\pm}0.3$	$97.5{\pm}2.3$	$72.3{\pm}0.5$	$70.0{\pm}3.1$	$73.3{\pm}2.5$	82.02
Qwen3-4B	TokenSkip	$97.0{\pm}0.3$	$97.5{\pm}2.3$	$71.7{\pm}0.5$	$70.0{\pm}3.1$	$73.3{\pm}3.1$	81.90
Qwen3-4B	SCA	$97.2{\pm}0.1$	$97.5{\pm}2.3$	$71.1{\pm}0.3$	$73.3{\pm}2.5$	$73.3{\pm}3.1$	82.49
Qwen3-8B	Base	$97.0{\pm}0.3$	$95.0{\pm}2.3$	$72.1{\pm}0.5$	$76.7{\pm}3.1$	$66.7{\pm}2.5$	81.49
Qwen3-8B	DeepCompress	$97.2{\pm}0.3$	$97.5{\pm}1.9$	$71.1{\pm}0.5$	$73.3{\pm}3.1$	$73.3{\pm}3.1$	82.49

Table 1: Pass@1 accuracy under multiple decoding seeds on mathematical benchmarks.

Average think length
Model	Method	MATH-500	AMC23	MinervaMath	AIME24	AIME25
Qwen3-4B	Base	3520	7333	3853	12215	15341
Qwen3-4B	DeepCompress	1861	4493	2555	8089	9758
Qwen3-4B	TokenSkip	2470	5150	2775	8750	11050
Qwen3-4B	SCA	1975	4831	2543	8192	9986
Qwen3-8B	Base	3211	6543	3309	9854	14064

Table 2: Average think and answer lengths on mathematical benchmarks.

Model	Method	Accuracy		Think Len.		Answer Len.
		HumanEval	GPQA	HumanEval	GPQA	HumanEval	GPQA
Qwen3-4B	Base	$87.80{\pm}0.80$	$58.08{\pm}0.66$	4017	7568	162	943
Qwen3-4B	SCA	$87.88{\pm}1.00$	$58.65{\pm}0.83$	3095	4921	153	895
Qwen3-8B	Base	$88.41{\pm}0.80$	$59.09{\pm}0.76$	2360	7814	176	874
Qwen3-8B	SCA	$88.49{\pm}1.00$	$59.09{\pm}0.66$	1702	2916	166	832

Table 3: Cross-domain accuracy, think length, and answer length on HumanEval and GPQA-Diamond.

Figure 4: Answer quality on final-answer-correct samples judged from the answer segment only. DC denotes DeepCompress.

Method	Acc.	Think	Answer
Base	93.8	1467	130
SCA	93.8	617	133
w/o Difficulty Weight	89.8	603	130
w/o Answer Alignment	93.8	579	79
w/o Segment Routing	93.3	1447	132
w/o Answer Length Reward	93.2	615	97

Table 4: Fine-grained component ablation on GSM8K-test.

Setting	Acc.	Think	Answer
Default: $f=32$ , $\alpha=0.5$	93.8	617	133
$f=64$	93.4	631	138
$\alpha=1.0$	94.3	654	129

Table 5: Hyperparameter diagnostics on GSM8K-test.

Setting	MATH	AMC	Min.	A24	A25
Base	3520	7333	3853	12215	15341
LoRA fixed	3440	7160	3788	11980	15020
LoRA cosine	3375	7055	3710	11840	14860

Table 6: LoRA transfer diagnostics across mathematical benchmarks.

Panel of Figure 5 — Figure 5: GSM8K-test think-length and answer-length distributions for the lightweight LoRA diagnostic at evaluation temperature 0.7.

CoT Compression

Chain-of-thought compression reduces the length or computation cost of model reasoning while aiming to preserve answer quality.

See all 57 papers

HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps
Jun 2, 2026Xin Liu, Runsong Zhao, Xinyu Liu +8Memory-Augmented Language Models Efficient Language Model Reasoning
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · China Unicom Cloud-Link, China · NiuTrans Research, Shenyang, China
Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression
Aug 31, 2026Tianyi Zhao, Yinhan He, Wendy Zheng +2LLM Compression Efficient Language Model Reasoning
University of Virginia
When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation
Jun 19, 2026Siyang Lyu, Xinghao Chen, Zhijing Sun +3Model Compression CoT Distillation
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Viterbi School of Engineering, University of Southern California · The Hong Kong Polytechnic University +2

Jun 2, 2026J/K move · Enter open · S save

HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps

Xin Liu, Runsong Zhao, Xinyu Liu, Junhao Ruan+7

School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · China Unicom Cloud-Link, China · NiuTrans Research, Shenyang, China

Memory-Augmented Language Models Efficient Language Model Reasoning Efficient Language Model Inference CoT Compression LLM Reasoning

Open paper AskPDF arXiv

Extended chain-of-thought (CoT) traces improve LLM reasoning but incur substantial computational and memory costs. While existing CoT compression methods mitigate this by condensing thought steps into compact representations via memory tokens and retaining only these representations at inference time, the loss of fine-grained information makes subsequent steps more error-prone. To alleviate this, we propose HybridThinker, where in addition to preserved these representations, thought steps are also temporarily retained to provide fine-grained details. However, we observe that naively keeping thought steps accessible to subsequent steps during training lets the model bypass memory tokens by retrieving information directly from these steps, leaving the model's ability to compress and retrieve information through memory tokens insufficiently trained. We therefore introduce a hybrid training scheme, in which only some thought steps are directly accessible through attention to subsequent steps, while the other thought steps are masked, forcing the model to use memory tokens for compression and retrieval. Across 4 reasoning benchmarks, HybridThinker matches the uncompressed baseline, advancing the state of the art in CoT compression by 5.8 points on average accuracy with similar inference time. Ablation studies confirm that both temporary thought-step retention and the hybrid training scheme contribute to these gains.

Abstract

CoT Compression

Explore similar work

HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps

Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression

When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation