Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectories
Authors: Peiyang Liu, Zhirui Chen, Xi Wang, Di Liang, Youru Li, Zhi Cai, Wei Ye
Organizations: National Engineering Research Center for Software Engineering, Peking University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Shanghai, China · Tencent Technology, Shenzhen, China · College of Computer Science, Beijing University of Technology, Beijing, China.
Monte Carlo Tree Search (MCTS) has been widely used for automated reasoning data exploration, but current supervision extraction methods remain inefficient. Standard approaches retain only the single highest-reward trajectory, discarding the comparative signals present in the many explored paths. Here we introduce Contrastive Reasoning Path Synthesis (CRPS), a framework that transforms supervision extraction from a filtering process into a synthesis procedure. CRPS uses a structured reflective process to analyze the differences between high- and low-quality search trajectories, extracting explicit information about strategic pivots and local failure modes. These insights guide the synthesis of reasoning chains that incorporate success patterns while avoiding identified pitfalls. We show empirically that models fine-tuned on just 60K CRPS-synthesized examples match or exceed the performance of baselines trained on 590K examples derived from standard rejection sampling, a 20× reduction in dataset size. Furthermore, CRPS improves generalization on out-of-domain benchmarks, demonstrating that learning from the contrast between success and failure produces more transferable reasoning capabilities than learning from success alone.
Figures & tables
Figure 1: Comparison between traditional MCTS selection methods and our CRPS framework. While standard methods discard suboptimal paths (grey), CRPS leverages the contrast between high-quality (green) and low-quality (red) trajectories to synthesize superior reasoning chains.
Figure 2: Overview of the CRPS framework. Given a problem, we first collect diverse trajectories from MCTS and stratify them into gold positive (green), soft negatives (blue), and hard negatives (red) groups based on terminal rewards. We then perform contrastive analysis at both trajectory-level (capturing strategic patterns) and step-level (capturing tactical patterns) to identify systematic differences between successful and unsuccessful reasoning. Finally, we synthesize novel reasoning paths by incorporating discovered success patterns while explicitly avoiding identified failure modes, producing a high-quality dataset with verified correctness.
Model
#Samples
In-Domain
Out-of-Domain
Avg.
MATH
GSM8K
College
DM
Olympiad
Theorem
DeepSeekMath-7B (Math-Specialized Base Model)
DeepSeekMath-Instruct
780K
46.9
82.7
37.1
52.2
14.2
28.1
43.5
DeepSeekMath-RFT
590K
53.0
88.2
41.9
60.2
19.1
27.2
48.3
DeepSeekMath-DART
590K
53.6
86.8
40.7
61.6
21.7
32.2
49.4
DeepSeekMath-MMIQC
2.3M
45.3
79.0
35.3
52.9
13.0
23.4
41.5
Table 1: Performance comparison across training methods and dataset scales on DeepSeekMath backbone. Arrows indicate accuracy changes relative to the strongest baseline (highlighted in blue). Best results in each data scale are in bold . ∗ indicate statistical significance at p<0.05 compared to the best baseline (calculated via paired t-test).
Base Model
Method
Size
MATH
GSM8K
Avg
DeepSeekMath
DeepScaleR
40K
55.1
83.9
69.5
OpenThought
89K
54.8
83.2
69.0
CRPS (Ours)
30K
56.3
84.8
70.6
LLaMA3-8B
DeepScaleR
40K
40.5
80.2
60.4
OpenThought
89K
41.2
80.8
61.0
CRPS (Ours)
30K
42.1
81.8
62.0
Table 2: Performance comparison against SOTA open-source SFT datasets.
Configuration
MATH
GSM8K
OOD
Overall
CRPS ( K=10 )
56.3
84.8
40.3
50.4
1. Impact of Synthesis Paradigm
Vanilla MCTS
52.1
81.5
38.4
47.7
Blackbox Generation
48.3
79.2
35.7
45.1
2. Necessity of Contrastive Signal
Synthesis w/o Contrast
51.7
82.3
37.8
47.5
Table 3: Ablation study results on DeepSeekMath-7B (30K scale).
Method
ID
OOD
Gap
Δ Gap
Breadth: In-Domain vs. Out-of-Domain Transfer
Vanilla MCTS
66.8
32.4
34.4
–
MathFusion
63.3
37.0
26.3
-4.6
SIGMA
68.6
38.0
30.6
-3.8
CRPS
70.6
40.3
30.3
-4.1
Depth: Difficulty-Stratified Performance
Table 4: Generalization analysis on DeepSeekMath-7B with 30K training examples.
Figure 3: An illustration of the reasoning performance of fully fine-tuned DeepSeekMath-7B models. The models are trained with data sizes ranging from 15K to 2.3M.
Model
#Samples
In-Domain
Out-of-Domain
Avg.
MATH
GSM8K
College
DM
Olympiad
Theorem
LLaMA3-8B (General-Purpose Base Model)
LLaMA3-MetaMath
400K
32.5
77.3
20.6
35.0
5.5
13.8
30.8
LLaMA3-RFT
590K
39.7
81.7
23.9
41.7
9.3
14.9
35.2
LLaMA3-DART
590K
46.6
81.1
28.8
48.0
14.5
19.4
39.7
LLaMA3-MMIQC
2.3M
39.5
77.6
29.5
41.0
9.6
16.2
35.6
Table 5: Performance comparison across base models, training methods, and dataset scales. Arrows indicate accuracy changes relative to the strongest baseline (highlighted in blue). Best results in each data scale are in bold . ∗ indicate statistical significance at p<0.05 compared to the best baseline (calculated via paired t-test).
Method
HumanEval
StrategyQA
Code Generation (Base: Qwen2.5-Coder-7B)
Supervised FT
48.2
–
Best-of-32 Sampling
52.4
–
Self-Refinement
54.1
–
CRPS (Ours)
57.9
–
Commonsense (Base: LLaMA3-8B)
Table 6: Performance on non-mathematical reasoning tasks. We report pass@1 accuracy for HumanEval and exact-match accuracy for StrategyQA. CRPS consistently improves performance over standard refinement methods by leveraging the contrast between valid and invalid execution traces.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Target
Dataset
Training
Acc.
Size
Cost (h)
Baselines
DART-Math
49.4%
590K
192
RFT
48.3%
590K
192
SIGMA
48.2%
30K
10
Ours
Appendix
Table 7: Computational cost breakdown to achieve ≈ 50% accuracy on MATH+GSM8K (DeepSeekMath-7B). CRPS shifts the compute burden from massive exploration and training to targeted synthesis, resulting in a 19.2 × reduction in training GPU hours compared to standard search-based selection.
Priority
Delimiter Type
Specific Tokens / Patterns
Rationale
1
Structure (Primary)
\n , \n\n , \ ( L a T e X newline)
Natural boundaries in math derivations; ensures semantic completeness.
2
Logic Connectors
Therefore, , Thus, , Hence, , So, , Consequently,
Signals a deductive leap or conclusion; critical for value estimation ( Q(s) ).
3
Explicit Enumeration
1. , 2. , Step 1: , First,
Explicitly structured reasoning steps in CoT.
Appendix
Table 8: Heuristic rules for step segmentation in MCTS and CRPS semantic alignment.