Training large language models (LLMs) with synthetic reasoning data has become a popular approach to enhancing their reasoning capabilities, while a key factor influencing the effectiveness of this paradigm is the quality of the generated multi-step reasoning data. To generate high-quality reasoning data, many recent methods generate synthetic reasoning paths and filter them based on final answer correctness, often overlooking flaws in intermediate reasoning steps. To enhance the verification of intermediate reasoning steps, prior work primarily resorts to code execution or symbolic reasoning engines. However, code-based validation is restricted to code or mathematical tasks, and reasoning engines require a well-structured and complete context. As a result, existing methods fail to function effectively in natural language reasoning tasks that involve ambiguous or incomplete contexts. In these tasks, synthetic data still lack reliable checks for verifying each reasoning step. To address this challenge, we introduce ORACLE, a structured data generation framework inspired by syllogistic reasoning. ORACLE integrates the generative strengths of LLMs with symbolic supervision: the LLM produces step-wise reasoning contexts, while a symbolic reasoning engine verifies the validity of each intermediate step. By employing a unified prompting template to elicit modular reasoning chains, ORACLE enables fine-grained, step-level validation, facilitating the construction of high-quality multi-step reasoning data. Across six logical, factual, and commonsense reasoning benchmarks, our ORACLE consistently outperforms strong baselines on multiple models.
Figures & tables
Figure 1: An overview of two-stage training pipeline of our ORACLE. ORACLE adopts a two-stage training pipeline: Stage 1 employs few-shot prompting and template-based reasoning generation, followed by answer-based and format-based filtering; Stage 2 integrates symbolic reasoning via beam search to produce high-quality reasoning data for supervised fine-tuning(SFT) and Direct Preference Optimization(DPO).
Figure 2: An overview of our structured reasoning template used during data generation of our ORACLE. Each reasoning step consists of modular fields: <QUERY> , <FACTS> , <RULE> , <REVISION> , <REVISION_RESULT> , and <REASONING_RESULT> . This design promotes interpretable reasoning, enables symbolic verification, and facilitates automatic extraction via pattern matching.
Figure 3: An overview of the beam search integrating a symbolic reasoning engine and LLM-based evaluation of our ORACLE. NL means natural language and SL means symbolic language. For each step, the LLM generates a candidate reasoning step in natural language, which is then translated into symbolic form and passed to the reasoning engine for execution. Each candidate is scored based on execution success ( W1 ), LLM’s precision assessment ( W2 ), and feasibility estimation ( W3 ). Candidates that are successfully executed receive a final score of W1+W3 , while others are scored as W2+W3 . The top- K candidates based on these scores are selected and expanded in the next layer. Complete reasoning paths that produce correct final answers are collected as supervised fine-tuning(SFT) data. Additionally, preference pairs are constructed by comparing symbolically validated nodes with their invalid siblings for Direct Preference Optimization(DPO).
Models
Methods
ProntoQA
ProofWriter
BoolQ
CosmosQA
ScienceQA
StrategyQA
Llama-3.1-8B-Instruct
CoT(zero-shot) ( Wei et al. 2022 )
94.3
53.7
85.0
74.2
90.5
89.9
CoT(four-shot) ( Wei et al. 2022 )
92.4
60.3
83.2
79.5
91.9
88.6
RFT ( Yuan et al. 2023 )
95.0
62.0
86.3
80.4
92.4
89.9
ToT-SFT ( Yao et al. 2023 )
93.6
57.0
82.5
74.7
90.7
85.2
self-rewarding ( Yuan et al. 2024 )
95.8
63.4
86.2
81.5
92.2
90.8
ORACLE(Ours)
97.2
67.5
87.7
82.4
91.7
91.8
Table 1: Comparison of various reasoning and data generation methods across six datasets and three models. Our proposed method consistently achieves the best or near-best performance on all datasets and models, demonstrating its effectiveness across symbolic, factual, and commonsense reasoning tasks.
Models
Methods
ProntoQA
ProofWriter
BoolQ
CosmosQA
ScienceQA
StrategyQA
Llama-3.1-8B-Instruct
w/o engine
96.1
67.0
87.9
81.9
90.5
89.9
w/o beam search
96.5
66.7
87.6
82.2
91.2
90.5
w/o DPO
96.6
67.2
87.6
82.0
91.4
91.1
ORACLE(Ours)
97.2
67.5
87.7
82.4
91.7
91.8
Mistral-7B-Instruct-v0.3
w/o engine
88.0
63.8
86.1
78.3
82.9
88.8
w/o beam search
86.9
64.2
86.6
79.1
83.2
89.4
Table 2: Ablation study on key components of our framework. We compare our full method with three ablated variants to analyze the contribution of key components: w/o engine removes reasoning engine’s guidance during data generation; w/o beam search disables beam-based search while retaining the reasoning engine; w/o DPO omits Direct Preference Optimization and saves other components during training. While our full method does not always yield the best score on every dataset-model pair, it consistently achieves strong performance and often outperforms ablations across most settings, validating the effectiveness of combining symbolic guidance, beam search, and preference learning.
Models
Iterations
ProntoQA
ProofWriter
BoolQ
CosmosQA
ScienceQA
StrategyQA
Llama-3.1-8B-Instruct
1st iteration
82.7
52.3
45.2
55.2
30.3
24.4
2nd iteration
84.1
55.7
46.3
56.1
32.1
25.7
Mistral-7B-Instruct-v0.3
1st iteration
64.1
45.2
40.6
43.7
31.8
20.3
2nd iteration
66.3
46.1
41.2
44.0
32.5
21.0
Qwen-2.5-7B-instruct
1st iteration
57.5
48.7
33.1
47.8
25.5
32.8
2nd iteration
58.4
49.2
33.0
48.5
25.3
33.4
Table 3: The success rate of the reasoning steps executed by the reasoning engine in our ORACLE on six datasets. With more training iterations, the reasoning engine is able to successfully execute a greater proportion of reasoning steps.
Models
Error Reasons
ProntoQA
ProofWriter
BoolQ
CosmosQA
ScienceQA
StrategyQA
Llama-3.1-8B-Instruct
Generation Error
24.0
40.0
44.0
38.0
72.0
80.0
Translation Error
76.0
60.0
56.0
62.0
28.0
20.0
Table 4: Error type proportions of our ORACLE on Llama-3.1-8B-Instruct across six datasets when the reasoning engine fails to execute. Generation Error means that the proposed facts or rules are overly complex or incorrectly formatted to be effectively translated into symbolic language. Translation Error means that the limited translation capability of the LLM leads to syntactic or semantic errors in the symbolic representation.