Organizations: Singapore-MIT Alliance for Research and Technology · Korea Advanced Institute of Science and Technology · Massachusetts Institute of Technology · Singapore Management University
We present a method for generating training data for reinforcement learning with verifiable rewards to improve small open-weights language models on mathematical tasks. Existing data generation approaches rely on open-loop pipelines and fixed modifications that do not adapt to the model's capabilities. Furthermore, they typically operate directly on word problems, limiting control over problem structure. To address this, we perform modifications in a symbolic problem space, representing each problem as a set of symbolic variables and constraints (e.g., via algebraic frameworks such as SymPy or SMT formulations). This representation enables precise control over problem structure, automatic generation of ground-truth solutions, and decouples mathematical reasoning from linguistic realization. We also show that this results in more diverse generations. To adapt the problem difficulty to the model, we introduce a closed-loop framework that learns modification strategies through prompt optimization in symbolic space. Experimental results demonstrate that both adaptive problem generation and symbolic representation modifications contribute to improving the model's math solving ability.
Figures & tables
Figure 1: Left: Symbolic Data Generation with Prompt Optimization. We generate training data for mathematical tasks by modifying problems in symbolic space rather than directly in natural language. Starting from an original word problem (top left), we convert it to a symbolic representation (top right). An optimized prompt (middle, italics)—learned through closed-loop feedback from the model’s performance—specifies how to modify the symbolic structure to generate appropriately challenging problems. We call these prompts Opt-Sym . The modified symbolic representation (bottom right) is then translated back to a new word problem (bottom left). The LLM responses guide prompt optimization (green loop) to produce problems that are challenging. This symbolic approach enables precise control over problem structure and automatic ground-truth generation. In addition to introducing coupled equations, nested substitutions, note how the new word problem also returns a different story line than the original, i.e., generations through symbolic representations returns in more diverse training data. See Section 4.3 for a more detailed analysis. Right: Training on data generated via Opt-Sym improves model performance. We show the performance from training on data from Opt-Sym (green) compared to training on the original seed data (grey). The y-axis shows the performance averaged over several benchmarks, and the x-axis shows the amount of seed data used. Opt-Sym achieves an 8% improvement in performance even with as little as 100 seed data points. This is compared to the baseline’s 4% improvement. This demonstrates both the effectiveness and data efficiency of our approach. See Tables 1 and 2 and Figure 3 for the full results and analysis.
Figure 2: Top: Data Generation Pipeline. We generate new math problems from seed examples using two approaches. Left (Natural Language (NL) representation): Given a seed problem q∈Dseed , we prompt LLM Mgen to create a modified problem q′ , then prompt it again to generate its solution. Right (Symbolic Representation): Given a seed problem s∈Dsym , prompt Mgen to modify this representation to create s′ , translate s′ back to natural language to obtain q′ , and use a solver to compute the answer. Both pipelines produce problem-answer pairs for training. Bottom: Examples of optimized prompts and generated problems . Each column shows a different optimized prompt popt (top, truncated) applied to the same seed problem shown at the top. The resulting generated problem is show below. The blue text highlights the modifications requested by each prompt. These prompts are learned through closed-loop optimization (see Section 3.4 ) that takes student model performance as feedback, enabling them to target specific weaknesses. See Appendix D for complete optimized prompts across different settings. The optimized prompts for both natural language and symbolic representations suggest similar mathematical modifications (fractions, multi-step reasoning, additional variables), indicating some overlap in how problem difficulty should be increased for the student model. However, the actual generated problems exhibit distinct characteristics. The NL modifications produce problems with complex narratives that require careful interpretation, while symbolic modifications yield problems with explicit mathematical operations stated directly in the text. Thus, the space in which modifications are applied impacts the final problem structure and complexity.
Figure 3: Additional analysis. a. Data generation methods across subset sizes : The 1.5B model trained with Opt-Sym (green) outperforms the baseline approaches when trained on different data subsets. b. Data filtering impact : Filtering the generated training data improves performance on larger dataset sizes. This can be because for small datasets, filtering has a larger impact on data diversity, which can harm performance ( Setlur et al., 2024 ) . c. Prompt scaling : Performance increases monotonically with the number of prompts used during data generation. This indicates that having more diverse data helps improve performance (see Figure 4 for further analysis on data diversity). As we optimize for 4 prompts in total, there are (n4) options for selecting each of the n prompts etc. This shows the averaged accuracy (%) and standard deviation over the different combinations. d. Representation sensitivity : The choice of mathematical problem representation significantly affects model performance.
Data Type
Iter. 0
Iter. 1
Iter. 2
Iter. 3
Base-Sym
51.40
52.54
52.58
54.30
Opt-Sym
53.90
54.71
56.26
Table 3: Averaged accuracy (%) across multiple iterations of generation and fine-tuning on the 1.5B model. Opt-Sym shows consistent improvement as iterations increase. Base-Sym results with the same number of prompts per iteration are shown for comparison.
Figure 4: Analysis of generated data . Left: Generation via symbolic representations produce more diverse data. For each seed problem, we generate 10 variants using baseline and optimized prompts ( PNL and PSym ), compute their embeddings, and measure average pairwise cosine distance. Optimized prompts with symbolic representations (Sym) achieve substantially higher diversity than natural language (NL) variants. Right: Student models (1.5B and 3B) achieve lower accuracy on generated data compared to the seed data (dash-dot line), with optimized prompts producing the most challenging problems. Accuracy is similar across word-based and symbolic generation methods. The difficulty gaps from seed data suggests successful problem variations.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameters
GRPO
PPO
Training batch size
1024
1024
Samples per prompt
8
-
Max response length
1024
1024
Clip ratio
0.2
0.2
Scaling factor α
1.0
-
Training temperature
1.0
1.0
Appendix
Table 4: Full hyperparameter setting.
Figure 5: Performance vs different subsets of data for different RL methods and student models.
Data synthesis for training large reasoning models offers a scalable alternative to limited, human-curated datasets, enabling the creation of high-quality data. However, existing approaches face several challenges: (i) indiscriminate generation that ignores the solver's ability and yields low-value problems, or reliance on complex data pipelines to balance problem difficulty; and (ii) a lack of reasoning in problem generation, leading to shallow problem variants. In this paper, we develop a problem generator that reasons explicitly to plan problem directions before synthesis and adapts difficulty to the solver's ability. We recover problem-design CoT from human-authored multi-part questions, using later subquestions as targets to infer the design rationale. These traces initialize the generator, which is then optimized with a seed-conditioned reward based on solver answer consistency. The reward guides difficulty adjustment relative to each seed while favoring problems near the solver's competence boundary. Across 10 mathematical and general reasoning benchmarks, our method outperforms seed-only training by 2.69 and 3.39 percentage points after one and three rounds, respectively. The framework also extends to vision-language models. Our code is publicly available at https://github.com/WalkerWorldPeace/Reasoning-Synthesis.
Yongxian Wei, Yilin Zhao, Li Shen +7
Tsinghua University · Tencent · Sun Yat-sen University +1
Large Language Models (LLMs) demonstrate strong capabilities for solving scientific and mathematical problems, yet they struggle to produce valid, challenging, and novel problems - an essential component for advancing LLM training and enabling autonomous scientific research. Existing problem generation approaches either depend on expensive human expert involvement or adopt naive self-play paradigms, which frequently yield invalid problems due to reward hacking. This work introduces VHG, a verifier-enhanced hard problem generation framework built upon three-party self-play. By integrating an independent verifier into the conventional setter-solver duality, our design constrains the setter's reward to be jointly determined by problem validity (evaluated by the verifier) and difficulty (assessed by the solver). We instantiate two verifier variants: a Hard symbolic verifier and a Soft LLM-based verifier, with evaluations conducted on indefinite integral tasks and general mathematical reasoning tasks. Experimental results show that VHG substantially outperforms all baseline methods by a clear margin.
Yuhang Lai, Jiazhan Feng, Yee Whye Teh +1
Department of Data Science, City University of Hong Kong · School of Intelligence Science and Technology, Peking University · Work done during Jiazhan’s visiting at the Department of Statistics, University of Oxford. +2
Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity: for challenging problems, LLM fails to sample any correct trajectories, preventing RL from receiving meaningful positive feedback. At the same time, there often exist human-written reference solutions along with the problem (e.g., problems from AoPS), but directly fine-tuning on these solutions offers no benefit because models often cannot imitate human proofs that lie outside their own reasoning distribution. We introduce Reference-Guided Fine-Tuning (ReGFT), a simple and effective method that utilizes human-written reference solutions to synthesize positive trajectories on hard problems and train on them before RL. For each problem, we provide the model with a partial reference solution and let it generate its own reasoning trace, ensuring the resulting trajectories remain in the model's reasoning space while still benefiting from reference guidance. Fine-tuning on these reference-guided trajectories increases the number of solvable problems and produces a checkpoint that receives more positive rewards during RL. Across three benchmarks (AIME24, AIME25, BeyondAIME), ReGFT consistently improves supervised accuracy, accelerates DAPO training, and raises the final performance plateau of RL. Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.
Yangzhen Wu, Shanda Li, Zixin Wen +5
UC Berkeley · Carnegie Mellon University · ByteDance Seed