ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models

Andre He; Nathaniel Weir; Kaj Bostrom; Allen Nie; Darion Cassel; Sam Bayless; Huzefa Rangwala

cs.AIFeb 23, 2026

ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models

Authors: Andre He, Nathaniel Weir, Kaj Bostrom, Allen Nie, Darion Cassel, Sam Bayless, Huzefa Rangwala

Organizations: Carnegie Mellon University · Amazon Web Services

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising approach for training reasoning language models (RLMs) by leveraging supervision from verifiers. Although verifier implementation is easier than solution annotation for many tasks, existing synthetic data generation methods remain largely solution-centric, while verifier-based methods rely on a few hand-crafted procedural environments. In this work, we scale RLVR by introducing ReSyn, a pipeline that generates diverse reasoning environments equipped with instance generators and verifiers, covering tasks such as constraint satisfaction, algorithmic puzzles, and spatial reasoning. A Qwen2.5-7B-Instruct model trained with RL on ReSyn data achieves consistent gains across reasoning benchmarks and out-of-domain math benchmarks, including a 27% relative improvement on the challenging BBEH benchmark. Ablations show that verifier-based supervision and increased task diversity both contribute significantly, providing empirical evidence that generating reasoning environments at scale can enhance reasoning abilities in RLMs

Figures & tables

Figure 1: Overview of synthetic environment generation in the ReSyn data pipeline. An LLM is prompted with seed keywords to synthesize Python implementations of reasoning environments, each defining instance generation $\rho_{0}$ , observation $O$ , and reward $R$ . The generated environment is evaluated by an LLM judge. Ones that pass are added to the ReSyn dataset, while failed ones are revised with feedback and re-evaluated

Panel of Figure 2 — Figure 2: Example synthetic environment generated by the ReSyn pipeline.

Figure 3: Overview of the ReSyn training pipeline. Left (Instance Generation) : Each environment generates instances from $\rho_{0}$ , which are transformed by the observation function $O$ and reward function $R$ into questions and verifiers $(Q,V)$ . Right (Reinforcement Learning) : A policy model generates candidate solutions for $Q$ , which are evaluated by $V$ to provide rewards for model updates.

Model	#Params	BBH (zero-shot)	BBEH	GSM8K-test	AIME 2024
		mean@4	mean@4	mean@4	mean@128
Qwen-2.5-Instruct	7B	65.9	11.2	82.3	9.8
SynLogic*	7B	66.5	8.0	—	10.0
ReSyn	7B	75.2	14.3	91.4	14.0
Llama-3.1-Instruct	8B	44.7	8.2	70.7	3.5
Mistral-Instruct	7B	28.2	6.4	22.3	0.1

Table 1: Evaluation of ReSyn versus the base Instruct model across reasoning and math benchmarks. All benchmarks are evaluated in zero-shot conditions using temperature

0.8

and top-

p

0.95

sampling. *Performance of SynLogic-7B is taken as reported in Liu et al. (2025a) .

Model	0-shot	1-shot	3-shot
Instruct	$65.9\pm 0.5$	$67.1\pm 0.5$	$70.4\pm 0.5$
ReSyn	$75.2\pm 0.5$	$71.9\pm 0.5$	$73.2\pm 0.5$

Table 2: BigBench-Hard overall accuracy (%) under different prompting setups.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Embedding-Based semantic entropy of 1000-task samples from ReSyn vs SynLogic training data.

Explore similar work

Learning from Synthetic Data Improves Multi-hop Reasoning
Mar 2, 2026Anmol Kabra, Yilun Yin, Albert Gong +6RL for Language Model Reasoning Multi-Hop Reasoning
Cornell University · University of Cambridge · Stanford University
SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions
Apr 9, 2026Ashima Suvarna, Kendrick Phan, Mehrab Beikzadeh +2RL for Language Model Reasoning LLM Reasoning
University of California, Los Angeles
Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
Jun 8, 2026Han Zhou, Adam X. Yang, Laurence Aitchison +2Pairwise Preference Learning Reward Modeling
University of Cambridge · 2Mistral AI · *Work done as an AI Scientist Intern at Mistral AI.

Mar 2, 2026J/K move · Enter open · S save

Learning from Synthetic Data Improves Multi-hop Reasoning

Anmol Kabra, Yilun Yin, Albert Gong, Kamilė Stankevičiūtė+5

Cornell University · University of Cambridge · Stanford University

RL for Language Model Reasoning Multi-Hop Reasoning Compositional Reasoning

Open paper AskPDF arXiv

Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.

Abstract

Explore similar work

Learning from Synthetic Data Improves Multi-hop Reasoning

SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions

Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short