Large language models have demonstrated remarkable capabilities as general-purpose assistants, excelling in a wide range of reasoning tasks and supporting various aspects of daily web usage. This achievement represents a significant step toward achieving artificial general intelligence. Despite these advancements, the effectiveness of large language models often hinges on the specific prompting strategies employed, and there remains a lack of a robust framework to facilitate learning and generalization across diverse reasoning tasks. To address these challenges, we introduce a novel learning framework, Thought-Like-Pro. In this framework, we utilize imitation learning to imitate the Chain-of-Thought process which is verified and translated from reasoning trajectories generated by a symbolic Prolog logic engine. This framework proceeds in a prompt-guided but self-bootstrapped manner, that enables large language models to formulate rules and statements from given instructions and leverage the symbolic Prolog engine to derive results. Subsequently, large language models convert Prolog-derived successive reasoning trajectories into natural language chain-of-thought for imitation learning. The empirical findings indicate that our proposed approach greatly improves the reasoning capacity of large language models. By employing model averaging techniques, our method exhibits only a marginal decline in performance for distributional extrapolation tasks, showing robust generalization capabilities. We present a technical approach that integrates symbolic reasoning with language modeling, with the potential to support the development of large language models as cognitively inspired systems. The part of the dataset we used has been open-sourced.
Figures & tables
Fig. 1: Workflow overview of our Thought-Like-Pro .
Method
Acc ↑
GSM8K
ProofWriter
PrOntoQA
QASC
Average
Llama3-8B
Qwen3-8B
Llama3-8B
Qwen3-8B
Llama3-8B
Qwen3-8B
Llama3-8B
Qwen3-8B
Llama3-8B
Qwen3-8B
Prompt
8-shot, CoT
3-shot, CoT
2-shot
0-shot
–
Training-free baselines
Base model
79.60
89.81
53.70
57.95
53.60
62.50
95.60
94.20
70.63
76.12
Self-Consistency
80.10
90.25
61.20
63.70
87.60
86.40
96.00
95.80
81.23
84.04
TABLE I: Performance on Prolog-Tractable Reasoning Tasks.
Fig. 2: Performance Evolution with Varying Numbers of Reasoning Trajectories: Showing enhanced logical reasoning capabilities and performance improvements through increased trajectory iterations.
Fig. 3: Representative synthetic example for the financial logic reasoning scenario. Values and entities are synthetic and do not correspond to any proprietary data.
Fig. 4: Representative synthetic example for the security & compliance scenario. Values and entities are synthetic and do not correspond to any proprietary data.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
CSQA
SQA
Type
Base
Thought-Like-Pro
Base
Thought-Like-Pro
PM
14
16
18
20
II
27
17
25
15
HK
16
11
26
14
SH
22
12
12
8
AM
12
22
10
22
Appendix
TABLE V: ERROR DISTRIBUTION OF TOOL-FREE REASONING FAILURES
A common approach for teaching large language models (LLMs) to reason is to train on chain-of-thought (CoT) traces of in-distribution reasoning problems, but such annotated data is costly to obtain for every problem of interest. We want reasoning models to generalize beyond their training distribution, and ideally to generalize compositionally: combine atomic reasoning skills to solve harder, unseen reasoning tasks. We take a step towards compositional generalization of reasoning skills when addressing a target compositional task that has no labeled CoT data. We find that simply training models on CoT data of atomic tasks leads to limited generalization, but minimally modifying CoT formats of constituent atomic tasks to be composable can lead to improvements. We can train "atomic CoT" models on the atomic tasks with Composable CoT data and combine them with multitask learning or model merging for better zero-shot performance on the target compositional task. Such a combined model can be further bootstrapped on a small amount of compositional data using rejection sampling fine-tuning (RFT). Results on string operations and natural language skill compositions show that training LLMs on Composable CoT outperforms multitask learning and continued fine-tuning baselines within a given training data budget.
Fangcong Yin, Zeyu Leo Liu, Liu Leqi +2
The University of Texas at Austin · Princeton University
We show that tiny transformers can profitably employ a simple form of Chain of Thought, which we call protoreasoning, allowing us to study step-by-step reasoning on ~1M-parameter models and opening up opportunities for much more detailed experimentation and analysis than is feasible for larger models. Current Large Language Models exhibit impressive step-by-step reasoning, but we have yet to understand its generality, i.e., when and how LLMs learn genuinely general algorithms rather than "bags of heuristics." Such questions are hard to settle on compute-intensive frontier models trained on opaque data. To work at model scales far below the threshold for natural-language competence, we define reasoning-friendly tasks on Dyck languages (sentences of correctly nested brackets). We find that protoreasoning traces substantially close the out-of-distribution generalization gap, and ablations confirm that the trace's content, not merely its extra tokens, drives the gain.
The success of large language models (LLMs) across diverse NLP tasks has elevated the importance of reasoning chain optimization as a critical step in aligning model behavior with task objectives. Existing reasoning chain tuning methods often rely on black-box heuristics or gradient-free search, which lack interpretability, generalization, and sample efficiency. In this work, we introduce Thoughts-as-Planning, a novel framework that formalizes reasoning chain optimization as a sequential decision-making process over a latent semantic space. We model the LLM as a partially observable environment and learn a latent world model that simulates the effect of reasoning chain edits on downstream outputs. A proximity-preserving embedding space is constructed to encode reasoning chain-response dynamics, enabling planning via gradient descent or reinforcement learning. Our method supports multi-scale abstraction, allowing reasoning chain edits at token, segment, and instruction levels to be integrated into a unified planner. Through extensive experiments on language understanding and generation tasks, we demonstrate that Thoughts-as-Planning outperforms state-of-the-art reasoning chain tuning baselines in efficiency, robustness, and generalization, while offering interpretability through its structured planning trajectory. Our code is available at https://github.com/FastLM/Thoughts-as-Planning.
Dong Liu, Yanxuan Yu, Ying Nian Wu
University of California, Los Angeles · Columbia University