cs.LGMar 2, 2026

Learning from Synthetic Data Improves Multi-hop Reasoning

Authors: Anmol Kabra, Yilun Yin, Albert Gong, Kamilė Stankevičiūtė, Dongyoung Go, Johann Lee, Katie Z. Luo, Carla P. Gomes, +1 more

Organizations: Cornell University · University of Cambridge · Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.

Figures & tables

Figure 1: Synthetic-to-real transfer by learning knowledge composition. We RL fine-tune LLMs on rule-generated synthetic datasets (left). Despite containing no real-world knowledge, training on these datasets teaches models to compose knowledge (center). This fundamental skill of chaining information across multiple steps transfers to real-world multi-hop reasoning benchmarks (right).
Figure 2: F1 scores on real-world multi-hop reasoning datasets of LLMs RL fine-tuned on synthetic datasets. We observe consistent transfer from synthetic data training to real-world multi-hop reasoning benchmarks: {\{ RG-Family, RG-Knights, GSM- ∞\infty , PhantomWiki }→\text{PhantomWiki }\}\rightarrow {\{ HotpotQA, 2WikiMultihopQA, MuSiQue, CofCA, SynthWorlds-RM }\text{SynthWorlds-RM }\} . The performance transfer trends hold across model families and sizes (Qwen and Phi LLMs in 0.6-4B parameter range). On each synthetic dataset, we fine-tune each LLM with 2 random training seeds, and evaluate final checkpoints of both experiment runs. With this we calculate the standard error, shown as error bars. See Figure 6 for similar plots for larger LLMs Qwen3-4B and Qwen2.5-7B-Instruct.
  HotpotQA2WikiMultihopQAMuSiQue
Qwen3-0.6Bbase0.36±0.020.36\pm 0.020.37±0.020.37\pm 0.020.14±0.010.14\pm 0.01
format0.38±0.020.38\pm 0.020.34±0.020.34\pm 0.020.13±0.010.13\pm 0.01
Qwen3-1.7Bbase0.59±0.020.59\pm 0.020.64±0.020.64\pm 0.020.34±0.020.34\pm 0.02
format0.64±0.020.64\pm 0.020.67±0.020.67\pm 0.020.35±0.020.35\pm 0.02
Phi-4-mini-reasoningbase0.48±0.020.48\pm 0.020.66±0.020.66\pm 0.020.27±0.020.27\pm 0.02
format0.47±0.020.47\pm 0.020.48±0.020.48\pm 0.020.26±0.020.26\pm 0.02
Table 1: Ablation study on fine-tuning with binary format reward. F1 scores of Qwen3 and Phi-4-mini-reasoning LLMs do not improve when RL fine-tuned with binary reward for using <answer>...</answer> . Qwen2.5-1.5B-Instruct improves remarkably with binary format reward. We report standard error over the evaluation datasets.
Figure 3: Performance of intermediate training checkpoints on unseen test sets, stratified by question complexity. We evaluate Qwen3-0.6B and Qwen3-1.7B intermediate checkpoints trained on PhantomWiki (left) and GSM- ∞\infty (right) on held-out test sets. Test sets share no factual overlap with training data. Question complexity is defined as the number of document hops (PhantomWiki) or arithmetic operations (GSM- ∞\infty ) required to reach the answer. As fine-tuning progresses (lines growing darker), performance improves across all complexity levels, including out-of-domain (OOD) difficulties beyond those seen during training. See Figure 8 for similar plots for other LLMs.
 Accuracy on GSM- ∞\inftyF1 score on HotpotQA
 baseSFTRLbaseSFTRL
Qwen3-0.6B0.02410.77350.64520.35690.39950.4786
Qwen3-1.7B0.13540.77420.85320.59350.57610.6664
Table 2: Performance of SFT vs RL fine-tuning on synthetic (GSM- ∞\infty ) and real-world (HotpotQA) test sets. SFT on ground-truth solution traces from the GSM- ∞\infty training data improves performance on the synthetic task but not on the target HotpotQA. RL fine-tuning improves models on both tasks, indicating that it teaches transferable knowledge composition where SFT does not.
Figure 4: Groundedness of Qwen3-0.6B reasoning traces in MuSiQue and CofCA intermediate answers. For each dataset, we plot the fraction of model generations (reasoning traces) that contain the ground-truth n th intermediate answer. As RL fine-tuning progresses on PhantomWiki (left) and GSM- ∞\infty (right), reasoning traces include a higher proportion of correct intermediate answers. This indicates that RL fine-tuning with outcome-only reward on synthetic data also improves the reasoning process. See Figure 10 for similar analysis on other models.
Figure 5: Real-world multi-hop reasoning performance of Qwen3-0.6B intermediate training checkpoints. We evaluate checkpoints every 500 training steps, and report mean ±\pm standard error with the solid line and shaded region. Performance on all benchmarks improves steadily with training steps—equivalently, with the number of synthetic training samples—providing evidence for synthetic data scaling in RL fine-tuning.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: F1 scores on real-world multi-hop reasoning datasets of LLMs RL fine-tuned on synthetic datasets. Larger LLMs Qwen3-4B and Qwen2.5-7B-Instruct, which already perform strongly at initialization, show less consistent synthetic-to-real transfer. We note that memory constraints necessitated suboptimal hyperparameters for these models ( Section A.1 ), and scaling up RL fine-tuning for larger models remains an important direction. On each synthetic dataset, we fine-tune each LLM with a fixed training seed and evaluate the final checkpoint. We report the standard error of evaluating the final experiment checkpoints.
Figure 7: F1 scores on real-world multi-hop reasoning datasets of LLMs RL fine-tuned on training sets of real-world data. On each training dataset, we RL fine-tune each LLM with 2 random training seeds, following the same training setup as with synthetic datasets in Figure 2 (10K training samples, F1 scores as reward). RL fine-tuning on real-world data outperforms rule-generated synthetic data like PhantomWiki, which is expected as synthetic data lacks rich linguistic complexity of real-world data. In fact, in-domain training data outperforms other data sources, i.e., HotpotQA training data yields maximum gains for HotpotQA benchmark (light green beats others in the left most column), and similarly for 2WikiMultihopQA and MuSiQue. We report the standard error over both experiments and benchmark samples.
Figure 8: Performance of intermediate training checkpoints on unseen test sets, stratified by question complexity. We evaluate Qwen2.5-1.5B-Instruct and Phi-4-mini-reasoning intermediate checkpoints trained on PhantomWiki (left) and GSM- ∞\infty (right) on held-out test sets. Test sets share no factual overlap with training data. Question complexity is defined as the number of document hops (PhantomWiki) or arithmetic operations (GSM- ∞\infty ) required to reach the answer. As fine-tuning progresses (lines growing darker), performance improves across all complexity levels, including out-of-domain (OOD) difficulties beyond those seen during training.
Figure 9: Real-world multi-hop reasoning performance of LLM intermediate training checkpoints. We evaluate checkpoints every 500 training steps, and report mean ±\pm standard error with the solid line and shaded region. Performance of Qwen3-1.7B and Phi-4-mini-reasoning on all benchmarks improves steadily with training steps—equivalently, with the number of synthetic training samples—providing evidence for synthetic data scaling in RL fine-tuning. Qwen2.5-1.5B-Instruct performance saturates quickly.
Figure 10: Groundedness of LLM reasoning traces in MuSiQue and CofCA intermediate answers. For each dataset, we plot the fraction of model generations (reasoning traces) that contain the ground-truth n th intermediate answer. As RL fine-tuning progresses on PhantomWiki (left) and GSM- ∞\infty (right), reasoning traces include a higher proportion of correct intermediate answers. This indicates that RL fine-tuning with outcome-only reward on synthetic data also improves the reasoning process. The effect is lower for larger LLMs on the CofCA benchmark.

Explore similar work

CardsList
Feb 23, 2026

ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising approach for training reasoning language models (RLMs) by leveraging supervision from verifiers. Although verifier implementation is easier than solution annotation for many tasks, existing synthetic data generation methods remain largely solution-centric, while verifier-based methods rely on a few hand-crafted procedural environments. In this work, we scale RLVR by introducing ReSyn, a pipeline that generates diverse reasoning environments equipped with instance generators and verifiers, covering tasks such as constraint satisfaction, algorithmic puzzles, and spatial reasoning. A Qwen2.5-7B-Instruct model trained with RL on ReSyn data achieves consistent gains across reasoning benchmarks and out-of-domain math benchmarks, including a 27% relative improvement on the challenging BBEH benchmark. Ablations show that verifier-based supervision and increased task diversity both contribute significantly, providing empirical evidence that generating reasoning environments at scale can enhance reasoning abilities in RLMs

Andre He, Nathaniel Weir, Kaj Bostrom +4

Carnegie Mellon University · Amazon Web Services

Synthetic Data Generation for Language ModelsRL for Language Model ReasoningReinforcement Learning with Verifiable Rewards
AskPDF
Sep 25, 2026

Self-Play Search Distillation for Large Language Model Reasoning

Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.

Lorenzo Molfetta, Wai-Chung Kwan, Giacomo Frisoni +5

University of Bologna · University of Edinburgh · Huawei Technologies R&D (UK) Ltd.

LLM ReasoningSynthetic Data Generation for Language ModelsLanguage Model Distillation
AskPDF
Mar 1, 2026

CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning

Large Language Models (LLMs) have recently exhibited remarkable reasoning capabilities, largely enabled by supervised fine-tuning (SFT)- and reinforcement learning (RL)-based post-training on high-quality reasoning data. However, reproducing and extending these capabilities in open and scalable settings is hindered by three fundamental data-centric challenges: (1) the cold-start problem, arising from the lack of seed datasets with detailed, long Chain-of-Thought (CoT) trajectories needed to initialize reasoning policies; (2) limited domain coverage, as most existing open-source reasoning datasets are concentrated in mathematics, with limited coverage of broader scientific disciplines; and (3) the annotation bottleneck, where the difficulty of frontier-level reasoning tasks makes reliable human annotation prohibitively expensive or infeasible. To address these challenges, we introduce CHIMERA, a compact synthetic reasoning dataset comprising 9K samples for generalizable cross-domain reasoning. CHIMERA is constructed with three key properties: (1) it provides rich, long CoT reasoning trajectories synthesized by state-of-the-art reasoning models; (2) it has broad and structured coverage, spanning 8 major scientific disciplines and over 1K fine-grained topics organized via a model-generated hierarchical taxonomy; and (3) it employs a fully automated, scalable evaluation pipeline that uses strong reasoning models to cross-validate both problem validity and answer correctness. We use CHIMERA to post-train a 4B Qwen3 model. Despite the dataset's modest size, the resulting model achieves strong performance on a suite of challenging reasoning benchmarks, including GPQA-Diamond, AIME 24/25/26, HMMT 25, and Humanity's Last Exam, approaching or matching the reasoning performance of substantially larger models such as DeepSeek-R1 and Qwen3-235B.

Xinyu Zhu, Yihao Feng, Yanchao Sun +5

Department of Computer Science, University of Virginia, Charlottesville, VA, USA · Apple, Cupertino, CA, USA · Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

LLM ReasoningSynthetic Data Generation for Language ModelsScientific Reasoning in Language Models
AskPDF