Shorter, but Still Trustworthy? An Empirical Study of Chain-of-Thought Compression

Lingjie Zeng; Xiaofan Chen; Yanbo Wang; Xiuying Chen

cs.CLApr 5, 2026

Shorter, but Still Trustworthy? An Empirical Study of Chain-of-Thought Compression

Authors: Lingjie Zeng, Xiaofan Chen, Yanbo Wang, Xiuying Chen

Organizations: Mohamed bin Zayed University of Artificial Intelligence

Abstract

Long chain-of-thought (Long-CoT) reasoning models have motivated a growing body of work on compressing reasoning traces to reduce inference cost, yet existing evaluations focus almost exclusively on task accuracy and token savings. Trustworthiness properties, whether acquired or reinforced through post-training, are encoded in the same parameter space that compression modifies. This means preserving accuracy does not, a priori, guarantee preserving trustworthiness. We conduct the first systematic empirical study of how CoT compression affects model trustworthiness, evaluating multiple models of different scales along three dimensions: safety, hallucination resistance, and multilingual robustness. Under controlled comparisons, we find that CoT compression frequently introduces trustworthiness regressions and that different methods exhibit markedly different degradation profiles across dimensions. To enable fair comparison across bases, we propose a normalized efficiency score for each dimension that reveals how naïve scalar metrics can obscure trustworthiness trade-offs. As an existence proof, we further introduce an alignment-aware DPO variant that reduces CoT length by 19.3% on reasoning benchmarks with substantially smaller trustworthiness loss. Our findings suggest that CoT compression should be optimized not only for efficiency but also for trustworthiness, treating both as equally important design constraints.

Figures & tables

Figure 1: Overview of the evaluation framework. Each compressed model is compared against its matched uncompressed baseline under a uniform protocol across three trustworthiness dimensions.

Base: DeepScaleR-1.5B-Preview
Paradigm	Model	HarmBench $\uparrow$	AgentHarm $\downarrow$	MMLU-ProX $\uparrow$	FaithEval $\uparrow$
—	Baseline	21.5	—	26.2	51.0
L1	L1-Qwen-1.5B-Exact	20.3	—	25.7	34.6
L1	L1-Qwen-1.5B-Max	20.8	—	28.2	36.6
BeConcise	—	21.5	—	23.7	48.8
Base: DeepSeek-R1-Distill-Qwen-1.5B

Table 1: Trustworthiness metrics across all evaluated model families. In the column headers,

\uparrow

indicates that higher values are better and

\downarrow

indicates that lower values are better. HarmBench reports refusal accuracy (%); AgentHarm reports harmful task completion rate (%), where a lower score indicates a safer model; MMLU-ProX reports multilingual accuracy (%); and FaithEval reports truthfulness (%). Gray rows denote uncompressed baselines. “—” indicates entries excluded under the benchmark scope described in Section 3.1 .

Figure 2: Trustworthiness–length Pareto frontiers for the Qwen3-8B family. Each panel plots one trustworthiness metric against mean CoT token count. The black polyline marks the Pareto frontier.

	HarmBench		MMLU-ProX		FaithEval
Method	$E$	$L$	$E$	$L$	$E$	$L$
Base: DeepScaleR-1.5B-Preview
Baseline	1.00	655	1.00	863	1.00	1125
L1-Exact	0.90	643	0.89	1003	0.78	395
L1-Max	0.92	673	1.13	915	0.85	409
BeConcise	0.75	1161	0.90	711	0.96	1032

Table 2: Composite efficiency scores

E

(Eq. 1 ;

\alpha\!=\!2,\beta\!=\!0.5

) and mean CoT token count

L

for all tabulated methods.

E>1

indicates net benefit; underlined entries mark

E<1

(net loss).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Paradigm	Base Models	Weights
L1	Distillation (RL)	1.5B, 7B, 8B	Author
BeConcise	Prompt eng.	1.5B, 7B, 8B	N/A
Thinkless	RL	1.5B	Author
TokenSkip	Training-free	8B	Reprod.
VeriThinker	Distillation	7B	Author
DPO	Preference-based	7B, 8B	Ours

Appendix

Table 3: Compression methods evaluated in this study. ”Author” indicates official pretrained weights, ”Reprod.” indicates our reproduction.

Category	Status	Criterion / reason
Official checkpoints on supported base models	Included	Evaluated directly with author-released weights (e.g., L1, Thinkless, VeriThinker, O1-Pruned).
Methods reproducible from public code but without released weights	Included	Reproduced when the public implementation was sufficiently specified for a stable run in our evaluation stack (e.g., TokenSkip, model merging).
Training-free interventions applicable to supported bases	Included	No retraining required, directly evaluated as inference-time baselines (e.g., BeConcise).
Our preference-based compression variant	Included	Trained and evaluated by us on DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B, and DeepSeek-R1-0528-Qwen3-8B to provide feasibility evidence for preference-based compression.
Methods without usable public artifacts	Excluded	Omitted when neither runnable checkpoints nor sufficiently detailed public artifacts were available at submission time.
Methods with insufficient reproduction detail or incompatible evaluation path	Excluded	Omitted when available materials did not support a stable and auditable reproduction under our common protocol.

Appendix

Table 4: Coverage policy for methods considered in this study. Our evaluation targets the currently available and reproducible subset of CoT compression methods rather than the full set of papers in the literature.

Llama Guard vs. human annotations
	Human unsafe	Human safe
Llama Guard unsafe	175	30
Llama Guard safe	20	288
Llama Guard vs. deepseek-v4-flash
	DeepSeek unsafe	DeepSeek safe
Llama Guard unsafe	178	27

Appendix

Table 5: Confusion matrices on the same 513-case safety-judgment subset. Rows are Llama-Guard labels; columns are the comparison labels.

	AIME24		AIME25		GPQA-Diamond
Method	Acc.	CoT tokens	Acc.	CoT tokens	Acc.	CoT tokens
Baseline	73.3	11236.37	60.0	13822.08	57.1	7478.44
DPO	70.0 ( $-3.3$ )	11738.69 ( $+4.5\%$ )	56.7 ( $-3.3$ )	14745.36 ( $+6.7\%$ )	63.1 ( $+6.0$ )	7290.50 ( $-2.5\%$ )

Appendix

Table 6: Hard reasoning accuracy (%) and mean CoT token count for Qwen3-8B. Parentheses in the DPO row give the accuracy-point change and relative CoT-token change from the baseline.

	$\alpha=1$			$\alpha=2$			$\alpha=3$
Method	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$
L1-Exact	0.910	0.949	0.990	0.778	0.811	0.846	0.665	0.693	0.723
L1-Max	0.923	0.980	1.040	0.778	0.826	0.877	0.656	0.696	0.739
BeConcise	0.848	0.806	0.766	0.777	0.738	0.701	0.711	0.676	0.642
TokenSkip	1.042	1.108	1.179	0.989	1.052	1.119	0.938	0.999	1.063
DPO	0.959	0.969	0.979	0.906	0.916	0.925	0.856	0.865	0.874

Appendix

Table 7: Sensitivity of

E_{\text{HB}}

(HarmBench) on Qwen3-8B to

(\alpha,\beta)

variation.

	$\alpha=1$			$\alpha=2$			$\alpha=3$
Method	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$
L1-Exact	1.210	1.343	1.490	1.252	1.390	1.542	1.296	1.438	1.596
L1-Max	1.220	1.375	1.549	1.244	1.402	1.580	1.269	1.430	1.611
BeConcise	1.080	1.102	1.124	1.130	1.154	1.177	1.183	1.208	1.233
TokenSkip	0.972	1.012	1.054	0.890	0.926	0.964	0.814	0.848	0.882
DPO	1.088	1.154	1.224	1.083	1.148	1.218	1.078	1.143	1.213

Appendix

Table 8: Sensitivity of

E_{\text{MX}}

(MMLU-ProX) on Qwen3-8B to

(\alpha,\beta)

variation.

	$\alpha=1$			$\alpha=2$			$\alpha=3$
Method	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$	$\beta\!=\!.3$	$\beta\!=\!.5$	$\beta\!=\!.7$
L1-Exact	1.153	1.234	1.320	1.202	1.286	1.376	1.253	1.341	1.434
L1-Max	1.158	1.273	1.400	1.165	1.280	1.407	1.171	1.287	1.414
BeConcise	0.971	0.973	0.975	0.940	0.942	0.944	0.911	0.912	0.914
TokenSkip	1.051	1.094	1.138	1.040	1.082	1.126	1.029	1.071	1.115
DPO	1.034	1.064	1.096	1.023	1.053	1.084	1.012	1.042	1.073

Appendix

Table 9: Sensitivity of

E_{\text{FE}}

(FaithEval) on Qwen3-8B to

(\alpha,\beta)

variation.

Benchmark	Direction stability	Pairwise stability	Mean Kendall $\tau$
HarmBench	41/45 (91.1%)	83/90 (92.2%)	0.844
MMLU-ProX	43/45 (95.6%)	83/90 (92.2%)	0.844
FaithEval	45/45 (100.0%)	86/90 (95.6%)	0.911
Overall	129/135 (95.6%)	252/270 (93.3%)	0.867

Appendix

Table 10: Stability relative to the default

(\alpha,\beta)=(2,0.5)

. Direction denotes whether

E

is above or below 1; pairwise stability counts the 10 method pairs per benchmark and configuration.

Benchmark	Exact set	Stable original members (inclusion frequency)
HarmBench	30.8%	Baseline 91.2%; TokenSkip 99.8%
AgentHarm	67.8%	BeConcise 77.7%; TokenSkip 100.0%
MMLU-ProX	76.6%	L1-Exact 91.7%; L1-Max 100.0%; BeConcise 84.7%
FaithEval	95.3%	L1-Exact 97.5%; L1-Max 100.0%

Appendix

Table 11: Pareto-frontier stability over 10,000 paired bootstrap replicates. Exact set is the frequency with which the complete original frontier-member set is recovered.

Figure 3: Trustworthiness–length Pareto frontiers across all model groups. Colors denote compression methods, marker size denotes model scale. The black polyline marks the Pareto frontier in each panel.

CoT Compression

Chain-of-thought compression reduces the length or computation cost of model reasoning while aiming to preserve answer quality.

See all 57 papers

Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability
Oct 2, 2026Samuel Lewis-Lim, Xingwei Tan, Mario Sanger +2Faithfulness of Language Model Reasoning Efficient Language Model Reasoning
School of Computer Science, University of Sheffield · AstraZeneca
When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation
Jun 19, 2026Siyang Lyu, Xinghao Chen, Zhijing Sun +3Model Compression CoT Distillation
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Viterbi School of Engineering, University of Southern California · The Hong Kong Polytechnic University +2
CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization
Nov 7, 2025Ziqian Bi, Yinzhi Wang, Tianyang Wang +6Efficient Language Model Reasoning LLM Inference Acceleration
1Purdue University, USA · 2Baruch College, City University of New York, USA · 3The Ohio State University, USA +6

Oct 2, 2026J/K move · Enter open · S save

Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability

Samuel Lewis-Lim, Xingwei Tan, Mario Sanger, Zhixue Zhao+1

School of Computer Science, University of Sheffield · AstraZeneca

Faithfulness of Language Model Reasoning Efficient Language Model Reasoning Faithfulness of Language Model Explanations Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning allows humans to inspect how large language models reach their answers, and oversee model behaviour. This reasoning comes at an increased inference cost, motivating efficient methods that train models to solve tasks using fewer tokens. However, a common concern is that such training may cause models to skip important reasoning steps, so the CoT no longer faithfully reflects the model's decision. It is unclear whether or when this occurs in practice, since different efficiency methods apply length pressure to models' CoT in distinct ways, and faithfully explaining a model's decision takes more tokens on some tasks than others. To understand these dynamics, we fine-tune a variety of models with three methods that apply length pressure differently, namely a fixed generation budget, a per-example length target, and a group-relative length reward. We evaluate how efficient reasoning affects CoT faithfulness (i.e., how well the CoT reflects model decisions on related inputs) and monitorability (i.e., whether the CoT reveals when input interventions alter the output). We find that it affects faithfulness and monitorability differently. Faithfulness falls in most settings, primarily because the trained models are less consistent. Monitorability is more robust, as models keep acknowledging the influence on their answer even when the CoT is much shorter.

Abstract

CoT Compression

Explore similar work

Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability

When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation

CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization