Learning to Solve Complex Problems via Dataset Decomposition

Wanru Zhao; Lucas Caccia; Zhengyan Shi; Minseon Kim; Weijia Xu; Alessandro Sordoni

cs.LGFeb 23, 2026

Learning to Solve Complex Problems via Dataset Decomposition

Authors: Wanru Zhao, Lucas Caccia, Zhengyan Shi, Minseon Kim, Weijia Xu, Alessandro Sordoni

Organizations: University of Cambridge · Microsoft Research · Mila - Quebec AI Institute

Abstract

Curriculum learning is a class of training strategies that organizes the data being exposed to a model by difficulty, gradually from simpler to more complex examples. This research explores a reverse curriculum generation approach that recursively decomposes complex datasets into simpler, more learnable components. We propose a teacher-student framework where the teacher is equipped with the ability to reason step-by-step, which is used to recursively generate easier versions of examples, enabling the student model to progressively master difficult tasks. We propose a novel scoring system to measure data difficulty based on its structural complexity and conceptual depth, allowing curriculum construction over decomposed data. Experiments on math datasets (MATH and AIME) and code generation datasets demonstrate that models trained with curricula generated by our approach exhibit superior performance compared to standard training on original datasets.

Figures & tables

Figure 1: Left: We recursively decompose a math example

(q,cot,a)

into a set of smaller problems (depth 2 in the figure). We first split the

cot

into steps, then create a question for each step and an associated concept tag. We then ask the teacher model to solve the question step-by-step. We verify the final answer by ensuring it is the same as the answer obtained without the ground-truth step in context. We then recursively apply this procedure until a stopping criterion. Right: We create a graph of tags, where dependency relation is given by the hierarchy in the decomposition tree. The graph of tags is used to quantify the difficulty of a generated sub-problem.

Figure 2: Left: We show the decomposition of the math problem on the top obtained with our method, along with the associated concept tags. The problems in the solid white boxes are the generated sub-problems. Right: The graph of concept tags, obtained by connecting the tags across the dataset examples.

Model	# data	MATH-500
Open-Weights Models
Qwen2.5-1.5B ( Qwen et al., 2024 )	N.A.	35.0
Qwen2.5-3B ( Qwen et al., 2024 )	N.A.	42.6
Llama-3-70B ( Dubey et al., 2024 )	N.A.	42.5
Mixtral-8x22B ( Jiang et al., 2024 )	N.A.	41.7
Gemma2-27B ( Team et al., 2024 )	N.A.	42.7

Table 1: Comparison of testing accuracy to LLMs on the MATH-500 benchmark.# data refers to the number of examples used for fine-tuning. ‡ We evaluate based on the checkpoint released at https://huggingface.co/Qwen/Qwen2.5-1.5B using lighteval ( Habib et al., 2023 ) .

Model	# data	AIME2025
Open-Weights Models
Qwen2.5-72B-Instruct ( Qwen et al., 2024 )	N.A.	15.0
S1-32B ‡ ( Muennighoff et al., 2025 )	1000	13.3
Close-Source Models
GPT-4o ( OpenAI et al., 2023 )	N.A.	7.6
Qwen3-4B-Base SFT on AIME2024

Table 2: Comparison of testing accuracy to LLMs on the AIME 2025 benchmark. # data refers to the number of examples used for fine-tuning. ‡ We evaluate based on the checkpoint released at https://huggingface.co/simplescaling/s1-32B without budget forcing.

Model	HumanEval pass@1
Open-Weights Models
Llama3-8B ( Dubey et al., 2024 )	33.5
Mistral-7B ( Jiang et al., 2024 )	29.3
Gemma2-9B ( Team et al., 2024 )	37.8
Qwen2.5-1.5B-Instruct SFT on Codeforces-CoTs
Base	$34.15\pm 3.71$

Table 3: Comparison of testing accuracy to LLMs on the HumanEval benchmark. # data refers to the number of examples used for fine-tuning.

Original Data	Decomposed Data
Problem: Simplify $\frac{3^{4}+3^{2}}{3^{3}-3}$ . Express your answer as a common fraction. Solution: The common factor of 3 in the numerator and the denominator can be factored out: $\frac{3^{4}+3^{2}}{3^{3}-3}=\frac{3(3^{3}+3)}{3(3^{2}-1)}=\frac{3^{3}+3}{3^{2}-1}$ . Now compute: numerator is $27+3=30$ , denominator is $9-1=8$ , so $\frac{30}{8}=\boxed{\frac{15}{4}}$ .	Generated Data with Lower Difficulty Difficulty Score: 4.0 Problem: What is the value of $3^{3}$ ? Solution: To solve for $3^{3}$ , we multiply 3 by itself three times: $3\times 3=9$ , then $9\times 3=27$ . So, $3^{3}=27$ . Tag: Exponentiation
	Generated Data of Medium Difficulty Difficulty Score: 10.0 Problem: What is the greatest common divisor (GCD) of the numbers 30 and 8? Solution: To find the greatest common divisor (GCD) of 30 and 8, we need to identify the largest number that divides both 30 and 8 without leaving a remainder. 1. List the factors of 30: - 1, 2, 3, 5, 6, 10, 15, 30 2. List the factors of 8: - 1, 2, 4, 8 3. Identify the common factors of 30 and 8: - The common factors are 1 and 2. 4. The greatest of these common factors is 2. Tag: GCD Calculation
	Generated Data with Higher Difficulty Difficulty Score: 18.0 Problem: Simplify the expression $\frac{3^{3}+3}{3^{2}-1}$ . Solution: We first compute the powers $3^{3}=27,3^{2}=9$ . Next, substitute into the expression: $\frac{27+3}{9-1}$ . Perform addition and subtraction $\frac{30}{8}$ . And simplify the fraction $\frac{30}{8}=\frac{15}{4}$ . Final Answer is $\boxed{\frac{15}{4}}$ . Tag: Fraction Simplification

Table 4: Examples of an original data sample from MATH training set and its generated data samples.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

By Domain			By Difficulty Level
Domain	#Total	#Sampled	Level	#Total	#Sampled
Algebra	1744	84	Level 1	566	26
Counting and Probability	771	36	Level 2	1348	65
Geometry	870	43	Level 3	1592	77
Intermediate Algebra	1295	63	Level 4	1690	81
Number Theory	869	41	Level 5	2304	111

Appendix

Table 5: Statistics of the original and sampled MATH dataset by subject domain (left) and by difficulty level (right). Samples were drawn uniformly at random within each group to ensure representative coverage across topics and difficulty tiers.

Figure 3: Concept dependency graph constructed during the AIME data decomposition process. Nodes represent mathematical concepts, and edges indicate prerequisite relationships between concepts.

Quintile	Difficulty Score Range
Quintile 1 (Easiest)	2.0 – 4.0
Quintile 2	4.0 – 4.0
Quintile 3	4.0 – 6.0
Quintile 4	6.0 – 10.0
Quintile 5 (Hardest)	10.0 – 20.0

Appendix

Table 6: Definition of difficulty quintiles based on concept dependency graph scores. Each quintile groups problems whose scores fall within the specified range.

Figure 4: Zero-shot performance of the Qwen3-4B-Base model across difficulty quintiles. Accuracy decreases as problem difficulty increases, validating our proposed difficulty metric.

Domain	#questions	Keywords
Equation Manipulation	17	Equation Manipulation, Equation Simplification, Equation Solving, Equation Subtraction, Linear Equation, Linear Equation in Two Variables, Linear Equations, Polynomial Equation Solving, Polynomial Simplification, Quadratic Equation, Rational Equation, Rational Equation Solving, Rearranging Equations, Simplifying Expressions, Simultaneous Equations, Simultaneous Equations Solving, Solving Rational Equations, Substitution, Subtraction, Variable substitution
Elimination Method	14	Elimination Method, Equation Solving: Isolating Variables, Linear Equation Solving, Solving Linear Equations, Substitution Method
Addition	12	Addition, Addition of Integers, Arithmetic Addition, Arithmetic Operations, Column Addition, Digit Sum, Fraction Addition, Integer Addition, Multiplication, Place Value Addition, Summation
Divisibility	11	Divisibility, Divisibility Rules, Division Property of Equality, Polynomial Division
Clearing Fractions	10	Clearing Fractions, Common Denominator Calculation, Fraction Multiplication, Fraction Simplification, Fraction Subtraction, Partial Fraction Decomposition, Reciprocal Calculation, Simplifying Fractions, Subtracting Fractions with Common Denominator
Arithmetic Sequence	7	Arithmetic Sequence, Arithmetic Sequence Formula, Arithmetic Subtraction, Counting Integers in an Arithmetic Sequence, Modular Arithmetic, Sum of a Sequence, Sum of an Arithmetic Series

Appendix

Table 7: Summary of Cluster Tags . Sample count represents the number of mathematical problems associated with each cluster tag. Keywords are derived from the tags within each cluster.

Domain	#questions	Keywords
Combinatorial Probability	100	Combinatorial Probability, Counting & Probability
Basic Counting Principle	46	Basic Counting Principle, Counting Principle, Counting Principles, Fundamental Counting Principle, Fundamental Principle of Counting, Multiplication Principle of Counting
Combination Calculation	36	Combination Calculation, Combination Enumeration, Combination Formula, Combination Selection, Combination Subtraction, Combination and Permutation Calculation, Combinations, Combinations Calculation, Combinations Formula, Counting Combinations
Factorial	31	Factorial, Factorial Calculation, Factorial Division, Factorial Expansion, Factorial Manipulation, Factorial Multiplication, Factorial Properties, Factorial Simplification, Factorial simplification
Combinatorial Counting	30	Combinatorial Counting, Combinatorial Enumeration, Combinatorial Exclusion, Combinatorial Reasoning, Combinatorics, Counting, Counting Arrangements, Counting Multiples, Counting Subsets
Binomial Coefficient	29	Binomial Coefficient, Binomial Coefficient Calculation, Binomial Coefficient Formula, Binomial Coefficient Multiplication, Binomial Coefficient Simplification, Binomial Coefficients, Binomial Coefficients Calculation, Binomial Expansion, Binomial Probability Formula, Binomial Theorem

Appendix

Table 8: Summary of Cluster Tags (Sample Count

\geq

3) . Sample count represents the number of mathematical problems associated with each cluster tag. Keywords are derived from the tags within each cluster.

Domain	#questions	Keywords
Probability Distribution	2	Probability Distribution
Range of Sums for Dice Rolls	2	Range of Sums for Dice Rolls, Sum of Two Dice Rolls
Uniform Probability Distribution	2	Uniform Probability Distribution
Properties of Platonic Solids	2	Properties of Platonic Solids, Symmetry of Platonic Solids
Division of Constants	2	Division of Constants, Division of constants
Permutations of Multisets	2	Permutations of Multisets

Appendix

Table 9: Summary of Cluster Tags (Sample Count

<

3) . Sample count represents the number of mathematical problems associated with each cluster tag. Keywords are derived from the tags within each cluster.

Curriculum Learning

Curriculum learning organizes training examples, tasks, or environments to match how a model learns.

See all 174 papers

Bidirectional Curriculum Generation: A Multi-Agent Framework for Data-Efficient Mathematical Reasoning
Mar 5, 2026Boren Hu, Xiao Liu, Boci Peng +4Curriculum Learning LLM Mathematical Reasoning
Zhejiang University · University of Macau · Peking University +3
Level Up: Defining and Exploiting Transitional Problems for Curriculum Learning
Mar 14, 2026Amogh Inamdar, Zhenwei Tang, Ashton Anderson +1Item Difficulty Estimation Curriculum Learning
Department of Computer Science Columbia University · Department of Computer Science University of Toronto
Learning to Reason with Curriculum II: Compositional Generalization
Jun 26, 2026Nived Rajaraman, Audrey Huang, Miroslav Dudik +3Curriculum Learning Automata Learning
Microsoft Research. · University of Illinois Urbana-Champaign. · Microsoft Research, New York, NY.

Mar 5, 2026J/K move · Enter open · S save

Bidirectional Curriculum Generation: A Multi-Agent Framework for Data-Efficient Mathematical Reasoning

Boren Hu, Xiao Liu, Boci Peng, Xinping Zhao+3

Zhejiang University · University of Macau · Peking University+3

Curriculum Learning LLM Mathematical Reasoning Efficient Language Model Training Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Enhancing mathematical reasoning in Large Language Models typically demands massive datasets, yet data efficiency remains a critical bottleneck. While Curriculum Learning attempts to structure this process, standard unidirectional approaches (simple-to-complex) suffer from inefficient sample utilization: they blindly escalate complexity even when foundational gaps persist, leading to wasted computation on unsolvable problems. To maximize the instructional value of every training sample, we introduce a novel Bidirectional Curriculum Generation framework. Unlike rigid trajectories, our multi-agent ecosystem mimics adaptive pedagogy to establish a closed feedback loop. It dynamically generates data by either complicating problems to challenge the model or, crucially, simplying them to repair specific reasoning failures. This mechanism ensures that the model consumes only the most effective data at any given stage. Grounded in the Optimal Pacing Theorem, our approach optimizes the learning trajectory, significantly outperforming baselines while achieving superior reasoning performance with substantially fewer instruction samples.

Abstract

Curriculum Learning

Explore similar work

Bidirectional Curriculum Generation: A Multi-Agent Framework for Data-Efficient Mathematical Reasoning

Level Up: Defining and Exploiting Transitional Problems for Curriculum Learning

Learning to Reason with Curriculum II: Compositional Generalization