Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiency

Xingshuai Huang; Derek Li; Bahareh Nikpour; Parsa Omidi

cs.CLMar 31, 2026

Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiency

Authors: Xingshuai Huang, Derek Li, Bahareh Nikpour, Parsa Omidi

Organizations: Huawei Technologies Canada, Canada · Huawei Noah’s Ark Lab, Canada

Abstract

Chain-of-Thought (CoT) prompting has significantly improved the reasoning capabilities of large language models (LLMs). However, conventional CoT often relies on unstructured, flat reasoning chains that suffer from redundancy and suboptimal performance. In this work, we introduce Hierarchical Chain-of-Thought (Hi-CoT), a structured reasoning paradigm specifically designed to address the challenges of complex, multi-step reasoning. Hi-CoT decomposes the reasoning process into hierarchical substeps by alternating between instructional planning and step-by-step execution. This decomposition enables LLMs to better manage long reasoning horizons and maintain logical coherence. Extensive evaluations across diverse LLMs and mathematical reasoning benchmarks show that Hi-CoT consistently improves average accuracy by 6.2% (up to 61.4% on certain models and tasks) while reducing reasoning trace length by 13.9% compared to CoT. We further show that accuracy and efficiency are maximized when models strictly adhere to the hierarchical structure. Our code is available at https://github.com/XingshuaiHuang/Hi-CoT.

Figures & tables

Figure 1: Accuracy (%) and average token length for Qwen3-8B model ( Yang et al., 2025a ) across multiple prompting methods on mathematical reasoning benchmarks. For Hi-CoT (correct format), results are computed only from responses that strictly follow the hierarchical structure.

Figure 2: Comparison of reasoning methods. (a) CoT generates step-by-step reasoning, (b) Plan-and-Solve first creates a global plan and then executes it at a coarse level, while (c) Hi-CoT enforces fine-grained alternating instruction–execution steps for more structured and controlled reasoning.

Model	Prompting	AIME24	AMC	MATH500	Minerva	OlympiadBench	Avg.
Qwen3-0.6B	Standard	0.0	21.7	56.2	18.0	20.7	23.3
	CoT	3.3	27.7	55.4	15.8	20.7	24.6
	Plan-and-Solve	3.3	26.5	56.4	16.9	20.7	24.8
	Hi-CoT (format-relaxed)	6.7	25.3	60.0	18.0	23.0	26.6
	Hi-CoT	3.3	33.7	57.0	20.2	23.4	27.5
Qwen3-1.7B	Standard	0.0	25.3	58.2	23.9	21.2	25.7

Table 1: Accuracy (%) across 13 models and three prompting strategies. Hi-CoT consistently outperforms baseline prompting methods across all five mathematical reasoning benchmarks.

Figure 3: Average response length (tokens) for multiple prompting methods. The rightmost panel shows the macro-average. Lower value generally indicates higher efficiency.

Figure 4: Comparative performance of Hi-CoT across aggregate and format-compliant outputs. Accuracy and average tokens are evaluated on four mathematical benchmarks for selected Qwen models, highlighting the performance gain when the hierarchical reasoning structure is strictly followed.

Explore similar work

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains
May 25, 2026Hui Xie, Jie Liu, Ziyue Qiao +1Efficient Language Model Reasoning LLM Inference Acceleration
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Cognitive Loop of Thought: Reversible Hierarchical Markov Chain for Efficient Mathematical Reasoning
Apr 8, 2026Jia-Chen Zhang, Yu-Jie Xiong, Zheng ZhouLLM Mathematical Reasoning CoT Reasoning
School of Computer Science and Technology, East China Normal University 3663 North Zhongshan Road, Shanghai, China · School of Electronic and Electrical Engineering, Shanghai University of Engineering Science 333 Longteng Road, Shanghai, China
Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs
Feb 28, 2026Jie Cao, Tianwei Lin, Juekai Lin +7Efficient Language Model Reasoning CoT Reasoning
Zhejiang University · Tencent

May 25, 2026J/K move · Enter open · S save

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains

Hui Xie, Jie Liu, Ziyue Qiao, Joaquin Vanschore

Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China

Efficient Language Model Reasoning LLM Inference Acceleration CoT Compression LLM Reasoning Latent Reasoning Latent CoT Reasoning

Open paper AskPDF arXiv

Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models (LLMs), but incurs high inference cost due to lengthy autoregressive traces. Existing latent reasoning methods offer a promis- ing alternative, yet they often treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed and thereby degrading reasoning accuracy. In this work, we propose Selective Latent Thinking (SLT), a reasoning paradigm that selectively compresses redundant reasoning spans into latent representations while preserving precision-critical spans as explicit CoT within the same reasoning trajectory. Our primary instantiation uses a lightweight decoder to anticipate a short upcoming reasoning span and applies confidence- based gating to determine the longest span that can be reliably compressed; the resulting selective compression policy is further optimized with trajectory-level reinforcement learning to balance answer correctness against reasoning cost. Ex- periments across four mathematical reasoning benchmarks show that which spans are compressed is critical: with the same compressor, compressing arbitrary spans reduces the GSM-Hard accuracy of Qwen3-4B from 54.7% to 17.3%, whereas keeping numerical tokens explicit retains 51.8%. SLT reduces explicit reasoning length by 58.4% with only a 2.8-point accuracy drop compared to explicit CoT on Llama-3.2-1B. Notably, the principle extends beyond this design: instantiated within existing latent reasoning models via a learned compression gate, it improves accuracy at equal reasoning length (+2.6 points over Latent-SFT) and at shorter length (+5.6 points over Latent-GRPO with 13% fewer tokens).

Abstract

Explore similar work

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains

Cognitive Loop of Thought: Reversible Hierarchical Markov Chain for Efficient Mathematical Reasoning

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs