cs.LGMar 2, 2026
SaveLearning from Synthetic Data Improves Multi-hop Reasoning
Organizations: Cornell University · University of Cambridge · Stanford University
Abstract
Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.
Figures & tables
Figure 1: Synthetic-to-real transfer by learning knowledge composition. We RL fine-tune LLMs on rule-generated synthetic datasets (left). Despite containing no real-world knowledge, training on these datasets teaches models to compose knowledge (center). This fundamental skill of chaining information across multiple steps transfers to real-world multi-hop reasoning benchmarks (right).
Figure 2: F1 scores on real-world multi-hop reasoning datasets of LLMs RL fine-tuned on synthetic datasets. We observe consistent transfer from synthetic data training to real-world multi-hop reasoning benchmarks: RG-Family, RG-Knights, GSM- , HotpotQA, 2WikiMultihopQA, MuSiQue, CofCA, . The performance transfer trends hold across model families and sizes (Qwen and Phi LLMs in 0.6-4B parameter range). On each synthetic dataset, we fine-tune each LLM with 2 random training seeds, and evaluate final checkpoints of both experiment runs. With this we calculate the standard error, shown as error bars. See Figure 6 for similar plots for larger LLMs Qwen3-4B and Qwen2.5-7B-Instruct.
| HotpotQA | 2WikiMultihopQA | MuSiQue | ||
|---|---|---|---|---|
| Qwen3-0.6B | base | |||
| format | ||||
| Qwen3-1.7B | base | |||
| format | ||||
| Phi-4-mini-reasoning | base | |||
| format |
Table 1: Ablation study on fine-tuning with binary format reward. F1 scores of Qwen3 and Phi-4-mini-reasoning LLMs do not improve when RL fine-tuned with binary reward for using <answer>...</answer> . Qwen2.5-1.5B-Instruct improves remarkably with binary format reward. We report standard error over the evaluation datasets.
Figure 3: Performance of intermediate training checkpoints on unseen test sets, stratified by question complexity. We evaluate Qwen3-0.6B and Qwen3-1.7B intermediate checkpoints trained on PhantomWiki (left) and GSM- (right) on held-out test sets. Test sets share no factual overlap with training data. Question complexity is defined as the number of document hops (PhantomWiki) or arithmetic operations (GSM- ) required to reach the answer. As fine-tuning progresses (lines growing darker), performance improves across all complexity levels, including out-of-domain (OOD) difficulties beyond those seen during training. See Figure 8 for similar plots for other LLMs.
| Accuracy on GSM- | F1 score on HotpotQA | |||||
|---|---|---|---|---|---|---|
| base | SFT | RL | base | SFT | RL | |
| Qwen3-0.6B | 0.0241 | 0.7735 | 0.6452 | 0.3569 | 0.3995 | 0.4786 |
| Qwen3-1.7B | 0.1354 | 0.7742 | 0.8532 | 0.5935 | 0.5761 | 0.6664 |
Table 2: Performance of SFT vs RL fine-tuning on synthetic (GSM- ) and real-world (HotpotQA) test sets. SFT on ground-truth solution traces from the GSM- training data improves performance on the synthetic task but not on the target HotpotQA. RL fine-tuning improves models on both tasks, indicating that it teaches transferable knowledge composition where SFT does not.
Figure 4: Groundedness of Qwen3-0.6B reasoning traces in MuSiQue and CofCA intermediate answers. For each dataset, we plot the fraction of model generations (reasoning traces) that contain the ground-truth n th intermediate answer. As RL fine-tuning progresses on PhantomWiki (left) and GSM- (right), reasoning traces include a higher proportion of correct intermediate answers. This indicates that RL fine-tuning with outcome-only reward on synthetic data also improves the reasoning process. See Figure 10 for similar analysis on other models.
Figure 5: Real-world multi-hop reasoning performance of Qwen3-0.6B intermediate training checkpoints. We evaluate checkpoints every 500 training steps, and report mean standard error with the solid line and shaded region. Performance on all benchmarks improves steadily with training steps—equivalently, with the number of synthetic training samples—providing evidence for synthetic data scaling in RL fine-tuning.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: F1 scores on real-world multi-hop reasoning datasets of LLMs RL fine-tuned on synthetic datasets. Larger LLMs Qwen3-4B and Qwen2.5-7B-Instruct, which already perform strongly at initialization, show less consistent synthetic-to-real transfer. We note that memory constraints necessitated suboptimal hyperparameters for these models ( Section A.1 ), and scaling up RL fine-tuning for larger models remains an important direction. On each synthetic dataset, we fine-tune each LLM with a fixed training seed and evaluate the final checkpoint. We report the standard error of evaluating the final experiment checkpoints.
Figure 7: F1 scores on real-world multi-hop reasoning datasets of LLMs RL fine-tuned on training sets of real-world data. On each training dataset, we RL fine-tune each LLM with 2 random training seeds, following the same training setup as with synthetic datasets in Figure 2 (10K training samples, F1 scores as reward). RL fine-tuning on real-world data outperforms rule-generated synthetic data like PhantomWiki, which is expected as synthetic data lacks rich linguistic complexity of real-world data. In fact, in-domain training data outperforms other data sources, i.e., HotpotQA training data yields maximum gains for HotpotQA benchmark (light green beats others in the left most column), and similarly for 2WikiMultihopQA and MuSiQue. We report the standard error over both experiments and benchmark samples.
Figure 8: Performance of intermediate training checkpoints on unseen test sets, stratified by question complexity. We evaluate Qwen2.5-1.5B-Instruct and Phi-4-mini-reasoning intermediate checkpoints trained on PhantomWiki (left) and GSM- (right) on held-out test sets. Test sets share no factual overlap with training data. Question complexity is defined as the number of document hops (PhantomWiki) or arithmetic operations (GSM- ) required to reach the answer. As fine-tuning progresses (lines growing darker), performance improves across all complexity levels, including out-of-domain (OOD) difficulties beyond those seen during training.
Figure 9: Real-world multi-hop reasoning performance of LLM intermediate training checkpoints. We evaluate checkpoints every 500 training steps, and report mean standard error with the solid line and shaded region. Performance of Qwen3-1.7B and Phi-4-mini-reasoning on all benchmarks improves steadily with training steps—equivalently, with the number of synthetic training samples—providing evidence for synthetic data scaling in RL fine-tuning. Qwen2.5-1.5B-Instruct performance saturates quickly.
Figure 10: Groundedness of LLM reasoning traces in MuSiQue and CofCA intermediate answers. For each dataset, we plot the fraction of model generations (reasoning traces) that contain the ground-truth n th intermediate answer. As RL fine-tuning progresses on PhantomWiki (left) and GSM- (right), reasoning traces include a higher proportion of correct intermediate answers. This indicates that RL fine-tuning with outcome-only reward on synthetic data also improves the reasoning process. The effect is lower for larger LLMs on the CofCA benchmark.