Recursive Harness Self-Improvement for Frontier Reasoning Data Synthesis

Wenlong Zhang; Zhengbo Jiao; Chenxu Zhang; Lekang Jiang; SiYuan Ma; Qituan Zhang; Guo Chen; Linfeng Zhang

cs.AIOct 2, 2026

Recursive Harness Self-Improvement for Frontier Reasoning Data Synthesis

Authors: Wenlong Zhang, Zhengbo Jiao, Chenxu Zhang, Lekang Jiang, SiYuan Ma, Qituan Zhang, Guo Chen, Linfeng Zhang

Organizations: EPIC Lab, Shanghai Jiao Tong University · University of Science and Technology of China · Shanghai University of Finance and Economics · Imperial College London ·

Abstract

Generating progressively harder reasoning problems requires synthesis procedures that adapt as the task distribution evolves. Existing task-level recursion reuses generated problems as seeds but leaves the construction harness unchanged. We present task-harness co-evolution, a framework for recursive harness self-improvement (RSI) in reasoning-data synthesis. Online self-improvement converts intermediate solver failures into reusable skills during generation. Post-task self-improvement revises skills, prompts, and workflows after each batch, adopting candidates only when they generate harder valid tasks within a bounded cost increase. Model weights and verification criteria remain fixed. Across mathematics, coding, and science, mean solver accuracy decreases from 100.0% to 54.8% over fourteen evolution rounds. Ablations show that combining both update schedules produces harder tasks than fixed-harness recursion or either schedule alone. The resulting data improves downstream SFT and GRPO performance. In particular, a 27B student fine-tuned on 10K synthesized mathematics examples achieves 62.5% mean-16 accuracy on APEX, competitive with selected frontier-model references. These results support adapting the synthesis harness alongside the tasks to generate increasingly challenging data with downstream training value.

Figures & tables

Figure 1: Three reasoning-data synthesis regimes. (a) Fixed-pipeline synthesis maps independent seeds to outputs through the same harness

H_{0}

. (b) Task-level recursion feeds each generated task back as the next seed, while construction skills and workflows stay fixed. (c) Our approach recursively updates both tasks and the harness. A meta-agent uses generation logs and evaluation feedback, together with human-defined goals and update constraints, to revise skills, prompts, and workflows. Evaluated revisions guide subsequent generation under fixed verification criteria. Snowflakes and flames denote fixed and evolving harnesses, respectively.

Figure 2: Harder tasks, useful training data. (a) DeepSeek-V4-Pro accuracy falls from 100.0% to 54.8% across 150 seed chains (50 per domain, equally weighted). The line tracks Passed; Failed is its complement; the y-axis is truncated at 50%. (b) SFT on 10K synthesized math examples yields 62.5% APEX accuracy for a 27B student, competitive with selected frontier models. We evaluate all APEX models with 16 attempts per problem; the x-axis starts at 20%. APEX evaluation details: Appendix C .

Figure 3: Three-stage task–harness co-evolution. Stage 1 initializes screened seeds and $H_{0}$ . Stage 2 updates skills during generation, yielding $H_{n}^{\mathrm{online}}$ . Stage 3 processes feedback after batch completion or budget exhaustion and uses human inputs to propose a candidate (top). Candidate and online baseline each synthesize one round from shared current tasks under matched limits (bottom). Adopt the first passing candidate; three rejections (initial proposal plus two retries) retain the online snapshot and discard post-task edits. The gate requires lower solver accuracy with bounded cost growth. Model weights and verifier logic remain fixed.

Training data	APEX Shortlist	HMMT Feb 2025	LCB v6	SciBench	Overall
Qwen3.5-4B Thinking (Baseline Model)
No additional training	32.4	73.5	55.8	71.8	58.4
Static Data Synthesis
OpenMathInstruct-2	33.6 $\textstyle\uparrow$ 1.2	75.4 $\textstyle\uparrow$ 1.9	52.6 $\textstyle\downarrow$ 3.2	70.9 $\textstyle\downarrow$ 0.9	58.1 $\textstyle\downarrow$ 0.3
OpenCodeReasoning	27.5 $\textstyle\downarrow$ 4.9	70.8 $\textstyle\downarrow$ 2.7	58.7 $\textstyle\uparrow$ 2.9	71.6 $\textstyle\downarrow$ 0.2	57.2 $\textstyle\downarrow$ 1.2
WildSci	28.2 $\textstyle\downarrow$ 4.2	70.0 $\textstyle\downarrow$ 3.5	53.6 $\textstyle\downarrow$ 2.2	70.6 $\textstyle\downarrow$ 1.2	55.6 $\textstyle\downarrow$ 2.8

Table 1: GRPO with Qwen3.5-4B Thinking. Mean-16 accuracy (%; higher is better). Overall averages the four benchmarks; superscripts give percentage-point changes from the checkpoint. Changes are computed from the displayed scores. Bold marks column-best scores.

Figure 4: Difficulty progression. DeepSeek-V4-Pro accuracy (%; lower is harder) over (a) all 150 seed chains and (b–d) three domains of 50 seeds each. The shared y-axis spans 45–100%; full values are in Appendix D .

Figure 5: Update ablations. Four configurations on 50 shared seeds: fixed harness, online skill updates, post-task updates with candidate evaluation, and both schedules. Lower DeepSeek-V4-Pro accuracy indicates harder tasks; the y-axis starts at 45%.

Figure 6: Harness transfer and operator cost. (a) Frozen harnesses on 50 shared held-out mathematics seeds. (b) Per-round Extend/Revise/Verify costs divided by 50 reasoning seeds; other harness and search costs are excluded. Full values are in Appendix E .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3.8-27B configuration	APEX	Gain (pp)
Before SFT	56.3	—
SFT on our 10K math examples	62.5	+6.2

Appendix

Table 3: 27B training gain on APEX. Mean-16 accuracy (%). Scores are rounded to one decimal; gains are computed from the displayed scores.

Model	APEX
Kimi K3 (Think)	65.6
Ours: Qwen3.8-27B + SFT	62.5
DeepSeek-v4-pro	61.5
Gemini 3.1 Pro Preview	60.9
GPT-5.4 (xhigh)	54.2
DeepSeek-v4-flash-0731	44.8

Appendix

Table 4: APEX reference comparison. Mean-16 accuracy (%) on APEX 2025, rounded to one decimal. All scores are from our own evaluations using MathArena. Only our 27B model receives additional training in this comparison.

Domain	$r{=}0$	$1$	$2$	$3$	$4$	$5$	$6$	$7$	$8$	$9$	$10$	$14$
All (150)	100.0	97.7	92.8	89.2	85.5	81.5	77.3	72.8	68.3	64.3	59.7	54.8
Math (50)	100.0	98.5	91.0	87.5	84.0	80.0	75.5	70.0	65.5	62.0	57.0	50.0
Coding (50)	100.0	97.0	93.5	90.0	86.0	82.5	78.0	74.5	70.0	66.5	62.0	58.5
Science (50)	100.0	97.5	94.0	90.0	86.5	82.0	78.5	74.0	69.5	64.5	60.0	56.0

Appendix

Table 5: Difficulty progression. DeepSeek-V4-Pro accuracy (%; lower is harder) over the fixed seed cohort. The overall row is recomputed from the three equally weighted domains.

Configuration	$r{=}0$	$2$	$4$	$6$	$8$	$10$	$12$	$14$
Task-only	100.0	95.5	87.0	78.0	70.5	72.5	70.0	73.5
Online only	100.0	94.5	86.0	78.0	70.0	64.5	65.0	67.0
Post-task only	100.0	94.0	85.5	77.5	68.5	63.5	62.0	59.0
Full	100.0	91.0	84.0	75.5	65.5	57.0	54.0	50.0

Appendix

Table 6: Update ablations. DeepSeek-V4-Pro accuracy (%; lower is harder) on 50 shared seeds. Online only and Post-task only refer to the update mechanisms in Sections 3.2 and 3.3 , respectively. Bold marks the lowest non-initial accuracy in each column.

Harness	$r{=}0$	$2$	$4$	$6$	$8$	$10$
Initial $H_{0}$	100.0	95.5	87.0	78.0	70.5	70.0
Evolved $H_{14}$	100.0	90.0	83.5	70.0	62.5	52.0

Appendix

Table 7: Frozen-harness comparison. DeepSeek-V4-Pro accuracy (%; lower is harder) on the tasks generated from 50 shared held-out mathematics seeds. Both harnesses remain frozen.

Configuration	Measure	$r{=}2$	$4$	$6$	$8$	$10$	$12$	$14$
Task-only	Accuracy	95.5	87.0	78.0	70.5	72.5	70.0	73.5
	Cost	1.022	1.165	1.252	1.364	1.368	1.372	1.332
Full	Accuracy	91.0	84.0	75.5	65.5	57.0	54.0	50.0
	Cost	1.042	1.199	0.754	0.887	0.931	1.032	1.065

Appendix

Table 8: Accuracy and generation-operator cost. DeepSeek-V4-Pro accuracy (%) and per-round cost (USD per original seed) on 50 reasoning seeds. Lower is better for both measures.

Parameter	SFT (35B / 27B)	GRPO (4B)
Training hardware	$16\times$ NVIDIA H20	$16\times$ NVIDIA H20
Epochs	2	1
Learning rate	$10^{-5}$	$10^{-6}$
Training batch size	16 examples	32 prompts
Sequence limits	65,536 total	3,072 prompt; 65,536 response
Distributed strategy	FSDP2	FSDP

Appendix

Table 9: Downstream training hyperparameters. The SFT settings apply to both the 35B and 27B students. Sequence limits are token ceilings, not observed average lengths. Dashes denote values not specified here.

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 285 papers

Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis
Nov 13, 2025Yongxian Wei, Yilin Zhao, Li Shen +7Synthetic Data Generation Synthetic Data Generation for Language Models
Tsinghua University · Tencent · Sun Yat-sen University +1
SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning
Oct 8, 2026Wei Yang, Shawn Li, Yuehan Qin +8Self-Evolving Agents Synthetic Data Generation for Language Models
University of Southern California · The George Washington University · Georgia Institute of Technology
MathAgent: Adversarial Evolution of Constraint Graphs for Mathematical Reasoning Data Synthesis
Apr 13, 2026Zixiong Yu, Jun Rao, Guhan Chen +5Synthetic Data Generation Synthetic Data Generation for Language Models
Huawei Large Model Data Technology Lab · Tsinghua University · Harbin Institute of Technology, Shenzhen +1

Nov 13, 2025J/K move · Enter open · S save

Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis

Yongxian Wei, Yilin Zhao, Li Shen, Zixuan Hu+6

Tsinghua University · Tencent · Sun Yat-sen University+1

Synthetic Data Generation Synthetic Data Generation for Language Models LLM Mathematical Reasoning

Open paper AskPDF arXiv

Data synthesis for training large reasoning models offers a scalable alternative to limited, human-curated datasets, enabling the creation of high-quality data. However, existing approaches face several challenges: (i) indiscriminate generation that ignores the solver's ability and yields low-value problems, or reliance on complex data pipelines to balance problem difficulty; and (ii) a lack of reasoning in problem generation, leading to shallow problem variants. In this paper, we develop a problem generator that reasons explicitly to plan problem directions before synthesis and adapts difficulty to the solver's ability. We recover problem-design CoT from human-authored multi-part questions, using later subquestions as targets to infer the design rationale. These traces initialize the generator, which is then optimized with a seed-conditioned reward based on solver answer consistency. The reward guides difficulty adjustment relative to each seed while favoring problems near the solver's competence boundary. Across 10 mathematical and general reasoning benchmarks, our method outperforms seed-only training by 2.69 and 3.39 percentage points after one and three rounds, respectively. The framework also extends to vision-language models. Our code is publicly available at https://github.com/WalkerWorldPeace/Reasoning-Synthesis.

Abstract

Synthetic Data Generation for Language Models

Explore similar work

Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis

SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning

MathAgent: Adversarial Evolution of Constraint Graphs for Mathematical Reasoning Data Synthesis