cs.CLOct 31, 2026

Enhancing Language Model Reasoning with Structured Multi-Level Modeling

Authors: Siheng Xiong, Ali Payani, Faramarz Fekri

Organizations: Georgia Institute of Technology · Cisco Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Inference-time scaling enhances a model's reasoning by extending its chain-of-thought (CoT). However, existing approaches typically rely on a single policy trained with outcome-reward reinforcement learning (RL), which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, especially for small LMs with limited capacity. To address this, we propose Multi-Level Reasoning (MLR), which reformulates long-CoT generation as a two-level stochastic process. A high-level planner generates structured step descriptors specifying both the reasoning mode and the semantic subgoal. The low-level executor then produces detailed reasoning conditioned on these descriptors, forming an alternating plan--execute loop. To maintain scalability, we adopt a minimal design where the base model serves as the low-level policy and a lightweight LoRA module implements the high-level policy. For training, we observe that outcome-reward RL provides sparse and delayed feedback for long trajectories (e.g., several thousand tokens), hindering credit assignment. We therefore introduce iterative Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision. This yields more effective training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks show that, under the same reduced data budget (10% SFT and 5% preference relative to the DeepSeek-R1 distillation setup), MLR outperforms both SFT-based distillation and strong RL/preference-optimization baselines across multiple base models and tasks. Moreover, MLR exhibits slower performance degradation on long-horizon reasoning, demonstrating stronger robustness under extended CoT generation.

Figures & tables

Figure 1: Long-horizon accuracy on MATH500, AIME24, and a mixture benchmark (MATH500 / AIME24 / GPQA / BoardGameQA). Top: Qwen-2.5-1.5B; bottom: Llama-3.1-8B. Problems are concatenated in the prompt to simulate long horizons; MLR degrades more slowly than baselines (see Appendix C for details).
Figure 2: The CoT trajectories can be long, and the position of the first error varies widely across trajectories, making outcome-only RL fine-tuning less informative. Statistics in (b) are computed from 100 trajectories with incorrect final answers, where the first error was manually identified.
Figure 3: Outcome-reward RL becomes increasingly costly for long trajectories: (a) PPO step latency grows with the maximum generation length, and (b) feedback is only available after the full trajectory completes. All measurements use Verl ( Sheng et al., 2024 ) with vLLM ( Kwon et al., 2023 ) on a single A100 node (batch size = 32).
Figure 4: Illustration of MLR. MLR augments single-policy reasoning with a high-level policy that generates step descriptors (reasoning mode and subgoal), providing structured guidance to reduce long-horizon plan drift and facilitate more effective learning.
Figure 5: Overview of the architecture. The model alternates between generating high-level descriptors and the corresponding low-level content. Ablations motivating the design are in Appendix B .
Figure 6: Monte Carlo rollout analysis. (a) Future-solvability estimation accuracy is similar between R1-Distill-LLaMA-8B and R1-Distill-Qwen-1.5B. (b) Runtime increases with model size, and R1-Distill-LLaMA-8B is substantially slower than R1-Distill-Qwen-1.5B. See Appendix B for implementation details.
Figure 7: Length distributions for high-level trajectories, low-level trajectories, and distilled summaries in our SFT dataset. Trajectories longer than 8,192 tokens are discarded.
 MathScienceLogic 
MethodMATH500AIME24GPQA (Diamond)BoardGameQA (Hard)Avg. Pass@1
 Pass@1Pass@1Cons@32Pass@1Pass@1 
Qwen-2.5-1.5B
Base29.6 ±\pm 0.60.0 ±\pm 0.00.0 ±\pm 0.021.2 ±\pm 1.135.0 ±\pm 1.821.5
Instruct54.6 ±\pm 0.43.3 ±\pm 1.510.0 ±\pm 0.325.2 ±\pm 1.441.0 ±\pm 1.031.0
SimpleRL59.0 ∗6.7 ∗4.2 ∗–––
Table 1: Overall performance across benchmarks. The best performance for each task using the same base model is in bold. Asterisks ( ∗ ) denote available results obtained from the official reports.
Figure 8: Agreement of selected preference pairs versus rollout count KK on (a) MATH500, (b) AIME24, (c) GPQA, and (d) BoardGameQA. For each KK , we report the fraction of rollout-selected pairs whose preference direction matches an 8B reference using 16 rollouts. We use the base model as prollp_{\text{roll}} for AIME24 and GPQA.
Figure 10
 MathScienceLogic 
MethodMATH500AIME24GPQA (Diamond)BoardGameQA (Hard)Avg. Pass@1
 Pass@1Pass@1Cons@32Pass@1Pass@1 
Ours91.5 ±\pm 1.353.2 ±\pm 2.073.3 ±\pm 0.452.8 ±\pm 1.567.0 ±\pm 1.466.1
DPO-only78.2 ±\pm 1.438.1 ±\pm 1.957.0 ±\pm 0.546.0 ±\pm 1.659.0 ±\pm 1.655.3
Low-level policy + Step-DPO82.4 ±\pm 1.342.6 ±\pm 2.061.2 ±\pm 0.549.2 ±\pm 1.562.1 ±\pm 1.459.1
Low-level policy + DPO74.1 ±\pm 1.532.4 ±\pm 1.852.0 ±\pm 0.644.0 ±\pm 1.756.0 ±\pm 1.751.6
Table 3: Ablation results for the core components of MLR using LLaMA-3.1-8B.
Figure 12
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13
Figure 14
Figure 13: Validation accuracy of Qwen2.5-1.5B SFT during GRPO training. For each question, we sample 8 responses and calculate the overall average accuracy to ensure a stable evaluation.
Figure 14: The average response length of Qwen2.5-1.5B SFT on the training set during GRPO.
Figure 15: Effect of rollout count on the reliability of utility estimates. We compare utilities estimated by the 1.5B model under KK rollouts to reference utilities by the 8B model using 16 rollouts for 100 sampled prefixes from MATH500 and AIME24, respectively.
Figure 16: Long-horizon reasoning response length on (a) MATH500, (b) AIME24, and (c) Mixture benchmarks (MATH500, AIME24, GPQA, BoardGameQA). We simulate long-horizon reasoning by concatenating multiple problems in the prompt and report average accuracy. Compared with R1-Distill models, which are prone to getting lost in long trajectories and entering dead loops, MLR benefits from explicit planning and is able to solve all problems more effectively.
More in this topic

Structured Reasoning in Language Models

See all 64 papers

Explore similar work

CardsList
Mar 31, 2026

Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiency

Chain-of-Thought (CoT) prompting has significantly improved the reasoning capabilities of large language models (LLMs). However, conventional CoT often relies on unstructured, flat reasoning chains that suffer from redundancy and suboptimal performance. In this work, we introduce Hierarchical Chain-of-Thought (Hi-CoT), a structured reasoning paradigm specifically designed to address the challenges of complex, multi-step reasoning. Hi-CoT decomposes the reasoning process into hierarchical substeps by alternating between instructional planning and step-by-step execution. This decomposition enables LLMs to better manage long reasoning horizons and maintain logical coherence. Extensive evaluations across diverse LLMs and mathematical reasoning benchmarks show that Hi-CoT consistently improves average accuracy by 6.2% (up to 61.4% on certain models and tasks) while reducing reasoning trace length by 13.9% compared to CoT. We further show that accuracy and efficiency are maximized when models strictly adhere to the hierarchical structure. Our code is available at https://github.com/XingshuaiHuang/Hi-CoT.

Xingshuai Huang, Derek Li, Bahareh Nikpour +1

Huawei Technologies Canada, Canada · Huawei Noah’s Ark Lab, Canada

CoT ReasoningLLM ReasoningEfficient Language Model Reasoning
AskPDF
May 4, 2026

Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding

Distilling large reasoning models is essential for making Long-CoT reasoning practical, as full-scale inference remains computationally prohibitive. Existing curation-based approaches select complete reasoning traces post-hoc, overlooking collaboration among heterogeneous teachers and lacking dynamic exploration, which leads to redundant sampling and missed complementary reasoning. We introduce CoRD, a collaborative multi-teacher decoding framework that performs step-wise reasoning synthesis guided by predictive perplexity-based scoring and beam search. This enables heterogeneous LRMs to jointly construct coherent reasoning trajectories while efficiently preserving diverse, high-potential hypotheses. Experiments show that CoRD produces higher-quality reasoning data and achieves near teacher-level student performance with fewer, structured supervision signals, without substantial efficiency overhead. CoRD further generalizes well to out-of-domain and open-ended settings. The dataset and model are available at https://github.com/DISL-Lab/CoRD.

Taewon Yun, Jisu Shin, Jeonghwan Choi +2

KAIST · UNIST

CoT DistillationEfficient Language Model ReasoningMulti-Teacher Knowledge DistillationTeacher-Student LearningMulti-LLM CollaborationLarge Reasoning Models
AskPDF
Mar 5, 2026

∇\nabla-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space

Scaling inference-time compute for Large Language Models (LLMs) has unlocked unprecedented reasoning capabilities. However, existing inference-time scaling methods typically rely on inefficient and suboptimal discrete search algorithms or trial-and-error prompting to improve the online policy. In this paper, we propose ∇\nabla-Reasoner, an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop to refine the policy on the fly. Our core component, Differentiable Textual Optimization (DTO), leverages gradient signals from both the LLM's likelihood and a reward model to refine textual representations. ∇\nabla-Reasoner further incorporates rejection sampling and acceleration design to robustify and speed up decoding. Theoretically, we show that performing inference-time gradient descent in the sample space to maximize reward is dual to aligning an LLM policy via KL-regularized reinforcement learning. Empirically, ∇\nabla-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark, while reducing number of model calls by approximately 10-40% compared to strong baselines. Overall, our work introduces a paradigm shift from zeroth-order search to first-order optimization at test time, offering a cost-effective path to amplify LLM reasoning.

Peihao Wang, Ruisi Cai, Zhen Wang +4

The University of Texas at Austin · UC San Diego · TTIC +1

LLM ReasoningTest-Time OptimizationEfficient Language Model ReasoningInference-Time Language Model Alignment
AskPDF