Training large language models (LLMs) with synthetic reasoning data has become a popular approach to enhancing their reasoning capabilities, while a key factor influencing the effectiveness of this paradigm is the quality of the generated multi-step reasoning data. To generate high-quality reasoning data, many recent methods generate synthetic reasoning paths and filter them based on final answer correctness, often overlooking flaws in intermediate reasoning steps. To enhance the verification of intermediate reasoning steps, prior work primarily resorts to code execution or symbolic reasoning engines. However, code-based validation is restricted to code or mathematical tasks, and reasoning engines require a well-structured and complete context. As a result, existing methods fail to function effectively in natural language reasoning tasks that involve ambiguous or incomplete contexts. In these tasks, synthetic data still lack reliable checks for verifying each reasoning step. To address this challenge, we introduce ORACLE, a structured data generation framework inspired by syllogistic reasoning. ORACLE integrates the generative strengths of LLMs with symbolic supervision: the LLM produces step-wise reasoning contexts, while a symbolic reasoning engine verifies the validity of each intermediate step. By employing a unified prompting template to elicit modular reasoning chains, ORACLE enables fine-grained, step-level validation, facilitating the construction of high-quality multi-step reasoning data. Across six logical, factual, and commonsense reasoning benchmarks, our ORACLE consistently outperforms strong baselines on multiple models.
Figures & tables
Figure 1: An overview of two-stage training pipeline of our ORACLE. ORACLE adopts a two-stage training pipeline: Stage 1 employs few-shot prompting and template-based reasoning generation, followed by answer-based and format-based filtering; Stage 2 integrates symbolic reasoning via beam search to produce high-quality reasoning data for supervised fine-tuning(SFT) and Direct Preference Optimization(DPO).
Figure 2: An overview of our structured reasoning template used during data generation of our ORACLE. Each reasoning step consists of modular fields: <QUERY> , <FACTS> , <RULE> , <REVISION> , <REVISION_RESULT> , and <REASONING_RESULT> . This design promotes interpretable reasoning, enables symbolic verification, and facilitates automatic extraction via pattern matching.
Figure 3: An overview of the beam search integrating a symbolic reasoning engine and LLM-based evaluation of our ORACLE. NL means natural language and SL means symbolic language. For each step, the LLM generates a candidate reasoning step in natural language, which is then translated into symbolic form and passed to the reasoning engine for execution. Each candidate is scored based on execution success ( W1 ), LLM’s precision assessment ( W2 ), and feasibility estimation ( W3 ). Candidates that are successfully executed receive a final score of W1+W3 , while others are scored as W2+W3 . The top- K candidates based on these scores are selected and expanded in the next layer. Complete reasoning paths that produce correct final answers are collected as supervised fine-tuning(SFT) data. Additionally, preference pairs are constructed by comparing symbolically validated nodes with their invalid siblings for Direct Preference Optimization(DPO).
Models
Methods
ProntoQA
ProofWriter
BoolQ
CosmosQA
ScienceQA
StrategyQA
Llama-3.1-8B-Instruct
CoT(zero-shot) ( Wei et al. 2022 )
94.3
53.7
85.0
74.2
90.5
89.9
CoT(four-shot) ( Wei et al. 2022 )
92.4
60.3
83.2
79.5
91.9
88.6
RFT ( Yuan et al. 2023 )
95.0
62.0
86.3
80.4
92.4
89.9
ToT-SFT ( Yao et al. 2023 )
93.6
57.0
82.5
74.7
90.7
85.2
self-rewarding ( Yuan et al. 2024 )
95.8
63.4
86.2
81.5
92.2
90.8
ORACLE(Ours)
97.2
67.5
87.7
82.4
91.7
91.8
Table 1: Comparison of various reasoning and data generation methods across six datasets and three models. Our proposed method consistently achieves the best or near-best performance on all datasets and models, demonstrating its effectiveness across symbolic, factual, and commonsense reasoning tasks.
Models
Methods
ProntoQA
ProofWriter
BoolQ
CosmosQA
ScienceQA
StrategyQA
Llama-3.1-8B-Instruct
w/o engine
96.1
67.0
87.9
81.9
90.5
89.9
w/o beam search
96.5
66.7
87.6
82.2
91.2
90.5
w/o DPO
96.6
67.2
87.6
82.0
91.4
91.1
ORACLE(Ours)
97.2
67.5
87.7
82.4
91.7
91.8
Mistral-7B-Instruct-v0.3
w/o engine
88.0
63.8
86.1
78.3
82.9
88.8
w/o beam search
86.9
64.2
86.6
79.1
83.2
89.4
Table 2: Ablation study on key components of our framework. We compare our full method with three ablated variants to analyze the contribution of key components: w/o engine removes reasoning engine’s guidance during data generation; w/o beam search disables beam-based search while retaining the reasoning engine; w/o DPO omits Direct Preference Optimization and saves other components during training. While our full method does not always yield the best score on every dataset-model pair, it consistently achieves strong performance and often outperforms ablations across most settings, validating the effectiveness of combining symbolic guidance, beam search, and preference learning.
Models
Iterations
ProntoQA
ProofWriter
BoolQ
CosmosQA
ScienceQA
StrategyQA
Llama-3.1-8B-Instruct
1st iteration
82.7
52.3
45.2
55.2
30.3
24.4
2nd iteration
84.1
55.7
46.3
56.1
32.1
25.7
Mistral-7B-Instruct-v0.3
1st iteration
64.1
45.2
40.6
43.7
31.8
20.3
2nd iteration
66.3
46.1
41.2
44.0
32.5
21.0
Qwen-2.5-7B-instruct
1st iteration
57.5
48.7
33.1
47.8
25.5
32.8
2nd iteration
58.4
49.2
33.0
48.5
25.3
33.4
Table 3: The success rate of the reasoning steps executed by the reasoning engine in our ORACLE on six datasets. With more training iterations, the reasoning engine is able to successfully execute a greater proportion of reasoning steps.
Models
Error Reasons
ProntoQA
ProofWriter
BoolQ
CosmosQA
ScienceQA
StrategyQA
Llama-3.1-8B-Instruct
Generation Error
24.0
40.0
44.0
38.0
72.0
80.0
Translation Error
76.0
60.0
56.0
62.0
28.0
20.0
Table 4: Error type proportions of our ORACLE on Llama-3.1-8B-Instruct across six datasets when the reasoning engine fails to execute. Generation Error means that the proposed facts or rules are overly complex or incorrectly formatted to be effectively translated into symbolic language. Translation Error means that the limited translation capability of the LLM leads to syntactic or semantic errors in the symbolic representation.
Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.
Anmol Kabra, Yilun Yin, Albert Gong +6
Cornell University · University of Cambridge · Stanford University
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising approach for training reasoning language models (RLMs) by leveraging supervision from verifiers. Although verifier implementation is easier than solution annotation for many tasks, existing synthetic data generation methods remain largely solution-centric, while verifier-based methods rely on a few hand-crafted procedural environments. In this work, we scale RLVR by introducing ReSyn, a pipeline that generates diverse reasoning environments equipped with instance generators and verifiers, covering tasks such as constraint satisfaction, algorithmic puzzles, and spatial reasoning. A Qwen2.5-7B-Instruct model trained with RL on ReSyn data achieves consistent gains across reasoning benchmarks and out-of-domain math benchmarks, including a 27% relative improvement on the challenging BBEH benchmark. Ablations show that verifier-based supervision and increased task diversity both contribute significantly, providing empirical evidence that generating reasoning environments at scale can enhance reasoning abilities in RLMs
Evaluating the reasoning abilities of large language models (LLMs) solely from final answers can obscure failures in intermediate steps, especially in multi-hop QA benchmarks without step-level annotations. To address this gap, we introduce Omanic, an open-domain 4-hop QA benchmark designed not only to measure final-answer accuracy but also to diagnose where reasoning breaks down. Omanic contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench), with each evaluation question decomposed into single-hop sub-questions, intermediate answers, and structured graph topologies. Experiments with proprietary and open-source LLMs show that Omanic is challenging, while step-wise analysis reveals a later-hop bottleneck, factual knowledge floor, and error propagation along reasoning chains. Fine-tuning on OmanicSynth transfers to six reasoning and mathematics benchmarks, yielding a 7.41-point average gain and validating its effectiveness as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.
Xiaojie Gu, Sherry T. Tong, Aosong Feng +8
The University of Tokyo · Yale University · Stanford University +3