Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

Shashank Aggarwal; Ram Vikas Mishra; Amit Awekar

cs.AIFeb 19, 2026

Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

Authors: Shashank Aggarwal, Ram Vikas Mishra, Amit Awekar

Organizations: Indian Institute of Technology Guwahati, Assam, India

Abstract

In multi-agent IR pipelines for tasks such as search and ranking, LLM-based agents exchange intermediate reasoning in terms of Chain-of-Thought (CoT) with each other. Current CoT evaluation narrowly focuses on target task accuracy. However, this metric fails to assess the quality or utility of the reasoning process itself. To address this limitation, we introduce two novel measures: reusability and verifiability. We decouple CoT generation from execution using a Thinker-Executor framework. Reusability measures how easily an Executor can reuse the Thinker's CoT. Verifiability measures how frequently an Executor can match the Thinker's answer using the CoT. We evaluated four Thinker models against a committee of ten Executor models across five benchmarks. Our results reveal that reusability and verifiability do not correlate with standard accuracy, exposing a blind spot in current accuracy-based leaderboards for reasoning capability. Surprisingly, we find that CoTs from specialized reasoning models are not consistently more reusable or verifiable than those from general-purpose LLMs like Llama and Gemma.

Figures & tables

Figure 1. Reusability Flowchart Flowchart showing how CoTs from correct answers of Thinker go to Executor and help/harm them to be counted towards reusability

Dataset	Executor	Thinker Model
Dataset	Executor	Gemma	Llama	Phi	R1
GSM8K	Full Comm.	66.04	73.46	83.53	66.56
	Strong Comm.	80.75	87.82	94.68	84.54
	Weak Comm.	51.33	59.10	72.37	48.59
StrategyQA	Full Comm.	42.13	43.35	45.25	44.58
	Strong Comm.	44.53	45.87	51.57	48.88

Table 1. Reusability Score across Datasets and Committees

Dataset	Executor	Thinker Model
Dataset	Executor	Gemma	Llama	Phi	R1
GSM8K	Full Comm.	79.46	71.33	88.36	80.33
	Strong Comm.	91.48	82.97	97.82	93.37
	Weak Comm.	67.45	59.70	78.91	67.29
StrategyQA	Full Comm.	57.05	44.98	58.52	61.64
	Strong Comm.	64.02	51.44	63.20	69.11

Table 2. Verifiability Score across Datasets and Committees

	Weak vs Strong	Strong vs Full	Full vs Weak
Dataset
GSM8K	0.6667	1.0000	0.6667
StrategyQA	-0.3333	1.0000	-0.3333
SVAMP	0.6667	1.0000	0.6667
ARC	0.6667	1.0000	0.6667
CommonSenseQA	1.0000	1.0000	1.0000

Table 3. Kendall’s

\tau

correlation of rankings based on reusability scores across weak, strong, and full committees.

	Weak vs Strong	Strong vs Full	Full vs Weak
Dataset
GSM8K	0.6667	1.0000	0.6667
StrategyQA	0.6667	0.6667	1.0000
SVAMP	0.6667	1.0000	0.6667
ARC	0.6667	0.6667	1.0000
CommonSenseQA	0.6667	0.6667	1.0000

Table 4. Kendall’s

\tau

correlation of rankings based on verifiability scores across weak, strong, and full committees.

Model Name	GSM8K			StrategyQA			SVAMP			ARC			CommonSenseQA
Model Name	A	R	V	A	R	V	A	R	V	A	R	V	A	R	V
Gemma3	$93$	$66$	$79$	$94$	$42$	$57$	$93$	$39$	$79$	$93$	$36$	$50$	$80$	$41$	$34$
Llama3.1	$78$	$73$	$71$	$71$	$43$	$45$	$82$	$65$	$72$	$83$	$43$	$47$	$68$	$43$	$37$
Phi4-Reasoning	$81$	$84$	$88$	$95$	$45$	$59$	$90$	$44$	$89$	$81$	$43$	$70$	$70$	$44$	$70$
DeepSeek-R1	$94$	$67$	$80$	$93$	$45$	$62$	$95$	$34$	$80$	$95$	$32$	$63$	$82$	$32$	$60$

Table 5. Model Performance Metrics across Multiple Datasets (Rounded to nearest integer)

Dataset	A vs R	R vs V	V vs A
GSM8K	-0.33	0.33	0.33
StrategyQA	0.33	0.33	0.33
SVAMP	-1.00	-0.33	0.33
ARC	-1.00	0.00	0.00
CommonSenseQA	-0.67	0.33	0.00
Average	-0.53	0.13	0.20

Table 6. Kendall

\tau

correlation between evaluation metrics (A=Accuracy, R=Reusability, V=Verifiability) across Datasets.

CoT Evaluation

Chain-of-Thought (CoT) evaluation measures the accuracy, validity, faithfulness, and usefulness of language models’ visible reasoning traces.

See all 21 papers

CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMs
Mar 7, 2026Siyi Li, Jiajun Shi, Shiwen Ni +9LLM Evaluation Efficient Language Model Reasoning
University of Science and Technology of China · Shenzhen Institutes of Advanced Technology, CAS · ByteDance SEED +6
TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
May 28, 2026Yundong Kim, Heyoung YangStructured Reasoning Language Model Generation Evaluation
Applied Agent Research Center, Korea Institute of Science and Technology Information (KISTI), Republic of Korea · Department of Computer Science and Engineering, University of Seoul, Republic of Korea.
CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning
May 19, 2026Dachuan Shi, Hanlin Zhu, Xiangchi Yuan +4Agentic Reasoning Efficient Language Model Inference
Georgia Tech · UC Berkeley · Stanford University +1

Mar 7, 2026J/K move · Enter open · S save

CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMs

Siyi Li, Jiajun Shi, Shiwen Ni, Ge Zhang+8

University of Science and Technology of China · Shenzhen Institutes of Advanced Technology, CAS · ByteDance SEED+6

LLM Evaluation Efficient Language Model Reasoning Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) have demonstrated strong performance by producing extended Chain-of-Thought (CoT) traces before answering. However, this paradigm often induces over-reasoning: redundant calculations and circular self-verification that increase computational cost without improving outcomes. Existing evaluations largely emphasize final accuracy or coarse token counts, and lack automated tools to separate essential logic from structural redundancy. We introduce CoTJudger, a graph-driven framework that quantifies reasoning efficiency by converting free-form CoTs into directed dependency graphs and extracting the Shortest Effective Path (SEP) needed to reach a correct solution. This yields an interpretable efficiency signal -- how much of a CoT is necessary versus structurally redundant -- that is comparable across models and tasks. Evaluating 21 LRMs, CoTJudger reveals pervasive redundancy and surfaces recurring failure modes, including verification obsession and compensatory redundancy. These results provide a practical metric for disentangling reasoning ability from computational waste, enabling more targeted evaluation and diagnosis of LRM efficiency.

Abstract

CoT Evaluation

Explore similar work

CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMs

TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation

CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning