cs.CLApr 9, 2026

Synthetic Data for any Differentiable Target

Authors: Tristan Thrush, Sung Min Park, Herman Brunborg, Luke Bailey, Marcel Roed, Neil Band, Christopher Potts, Tatsunori Hashimoto

Organizations: Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

What are the limits of controlling language models via synthetic training data? We develop a reinforcement learning (RL) primitive, the Dataset Policy Gradient (DPG), which can precisely optimize synthetic data generators to produce a dataset of targeted examples. When used for supervised fine-tuning (SFT) of a target model, these examples cause the target model to do well on a differentiable metric of our choice. Our approach achieves this by taking exact data attribution via higher-order gradients and using those scores as policy gradient rewards. We prove that this procedure closely approximates the true, intractable gradient for the synthetic data generator. To illustrate the potential of DPG, we show that, using only SFT on generated examples, we can cause the target model's LM head weights to (1) embed a QR code, (2) embed the pattern 67\texttt{67}, and (3) have lower ℓ2\ell^2 norm. We additionally show that we can cause the generator to (4) rephrase inputs in a new language and (5) produce a specific UUID, even though neither of these objectives is conveyed in the generator's input prompts. These findings suggest that DPG is a powerful and flexible technique for shaping model properties using only synthetic training examples.

Figures & tables

Figure 1: Dataset Policy Gradients allow us to generate synthetic training data for any differentiable target. For example, our generator can learn to generate special Wikipedia article rephrases. When used for continued pretraining of GPT-2, these rephrases turn the upper left 21x21 patch of GPT-2’s LM head weight matrix into the QR code seen here (when subtracted from the initial weights, sign’d, and visualized as a greyscale image). The text sample in this figure is the first item in the synthetic dataset, which we generated with a temperature of 1 (i.e., noisy data still produces the result).
Figure 2: Here, we initialize the target model in A\mathcal{A} to be GPT-2, and explore exotic target metrics: the goal of the first metric is to encode the greyscale image 67 in the upper 6x7 patch of the sign’d LM head weight updates to the target model. This number was chosen arbitrarily. The goal of the second metric is to lower the ℓ2\ell^{2} norm of the target model’s LM head. The plots show validation performance as the GRPO process trains the generator. All validations are done with 96 steps of continued training on GPT-2. The (96), (8), and (1) notation denotes whether the generator was trained via metagradients with respect to an A\mathcal{A} that used 96, 8, or 1 step(s). We observe a weak correlation between A\mathcal{A} steps and validation performance, and generally more validation stability with more A\mathcal{A} steps.
Figure 3: Final validation results for the 6x7 pixel images in the target models’ sign’d LM head updates, after the generator was fully trained. The numbers above the images denote the number of target model training steps in A\mathcal{A} for metagradient computation. All validations were done with 96 target model training steps, using the corresponding optimizer; the difference is whether the generator was trained using a reward function with fewer A\mathcal{A} training steps. Only Adam with 96 steps in A\mathcal{A} for metagrads achieved a generator that got a perfect result (we were close with the initial 96 run, so we trained the generator again with a different random sample of Wikipedia prompts – we then got a perfect score).
Figure 4: Generator results when setting Φ\Phi to be post-training loss on four multilingual LAMBADA ( Paperno et al., 2016 ) translations from Black (2023) : DE, ES, FR, and IT. We initialized the generator from Llama 3.2 Instruct. We initialized the target model in A\mathcal{A} also from Llama 3.2 Instruct. In each GRPO step, we conduct a single step of target model continued pretraining on the synthetic data before computing metagradients. When using Adam in A\mathcal{A} , the generator learns the correct language, as judged by GPT 4.1 Nano ( OpenAI, 2025 ) . Baselines do not learn the correct language except in rare cases where their entropy quickly collapses and they repeatedly produce only a few words.
Figure 5: We keep the same setup as the LAMBADA cases, with the exception of changing Φ\Phi to be the target model’s post-training LM loss on a 32-character UUID. In this plot, we show two validation metrics: Exact requires the complete UUID to be in a rollout, and Soft finds the longest substring of the UUID in the rollout and gives points proportional to the fraction of the UUID present.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: DPG RL, using GRPO. The target model in A\mathcal{A} is trained on generator rollouts. A\mathcal{A} ’s training loss incorporates weights for each training example. We compute gradients of the data weights with respect to some differentiable training or post-training target. We use these gradients as the rewards.
Figure 7: DPG RL, using GRPO. Same as Figure 6 , except we only conduct one large training run of A\mathcal{A} for each GRPO iteration, lumping all of the groups together. This is the approach we choose for nearly all of our experiments due to faster wallclock time and negligible influence on performance.
CPT Data SourceDEESFRITDEESFRIT
  Llama 3.2 Instr.  Llama 3.2 Base 
Before CPT133.86204.3189.23129.2693.12163.0165.1289.29
CPT on DCLM125.84209.5590.36133.4891.58160.5764.1087.55
Untuned Generator140.97218.4197.27145.0289.45144.8459.4682.79
Adam Metagrad64.0331.1233.0943.1335.0420.1818.5324.04
SGD Metagrad98.6553.6247.7586.8661.2533.5730.5653.74
Appendix
Table 1: Perplexity from the Eleuther Evaluation Harness ( Gao et al., 2024 ) of CPT’d models on our test split of the multilingual LAMBADA tasks. Rows designate the source of the CPT data. All CPT experiments are run with 10M tokens, which is far more than the single step case where our generators were optimized. Our DPG RL procedure with Adam in A\mathcal{A} is able to generate synthetic data that generalizes to this longer training regime, and is also able to generate data that generalizes to different models (it was optimized to generate data for Llama 3.2 Instruct in A\mathcal{A} , not Llama 3.2 Base; additionally, CPT for the final validation results here was done in PyTorch ( Ansel et al., 2024 ) , whereas the RL process used JAX ( Bradbury et al., 2018 ) ) for the target model. In the SGD case, we also see generalization, even though the Wikipedia rephrases are not the correct language (see Figure 4 ). Finally, we also present SFT results for reference, where we train on our LAMBADA train splits, and conduct a hyperparameter grid search with our LAMBADA val sets (for hyperparameters shown in Table 12 ).
GRPO Step, MMVal Generation
4Dakar Rally Co-Driver Timo Gottschalk: A Career Profile\n\n Born on August 28, 1974, in Neuruppin, Germany, Timo Gottschalk is a highly accomplished rally co-driver who has made a name for himself across the globe. What began as the beginning of a successful rally career at just 28 years old came easily for Gottschalk, who set out to pursue his passion for rallying at a relatively young age.\n\nEarly Career\n\nVoyaging toByteArray.round the ubiquitous Dakar Rally, Gottschalk’s first major road rally competition took place on July 10
16I’d be happy to help you with that! However, I don’t see an article provided. Please share the article you’d like me to read and rephrase, and I’ll do my best to assist you.\n\nIf you have the article, please paste it, and I’ll get started!
28Article Title: Timo Gottschalk: Priceless Years of Rallying in the World’s Most Popular Rally Event\n\nYear: 2023\n\n Title: Timo Gottschalk: A Firefighter and a Rally Legend\n\n Timo Gottschalk is a renowned motor racing co-driver renowned for being asMuch triumphant as Nasser Al-Attiyah in the 2011 Dakar Rally. Born on August 28 in Neuruppin, Gottschalk broke the heights of his fellow drivers with his achievements during the 2011 event, earning him the coveted title of win. As a seasoned driver
40I\’d be happy to help you read and rephrase the article in different terms! However, I need to correct some errors and clarify some points. Here\’s the rephrased article:\n\n**"Timo Gottschalk: A Renowned German Rally Co-driver"**\n\nTimo Gottschalk is a highly accomplished German co-driver who has made a name for himself as a innovative and experienced driver. Born on August 28, 1974, Gottschalk entered the world of rally driving at a young age, starting as an assistant in the 2007 German rally, where he worked with fellow co-driver Dieter De
Appendix
Table 2: 67-tuned Generator Rollouts for the first Wikipedia article in the validation set. By the last step (40), the generator is still producing coherent Wikipedia rephrases, but they end up encoding a secret message in the trained model weights.
GRPO Step, MMVal Generation
8Title: Of German Rally Driver Timo Gottschalk\n\nYear: 2011\n\nDriver / Driver Partner: Nasser Al-Attiyah, German rally co-driver for the 2011 Dakar Rally (occasionally a Doubles drive)\n\nTeam: Larsen-Welt Toyota Team ( whilst not including the engines used for the 2011 event possibly including a car though incorrectly stated once as Thor Wickinger, and Chassis, etc numbered modifications pen-and-red lines Bed totaling gun Mel/s donor numbers Then Drive-like match cells bunnylegs codying Shortoft**\n\n**Title Recognalias Relationship reduced Team
48The International Mozilla Handbook (published in 1974 by the see -and-dream-D-word-D.ralfllpaper constituted a comprehensive post-bwala Fusion and follow-hit automatic bombing para-flower assassination gr aynıVIN 189arsep:Document-name-psstarter San Opportunity Sun dismal Zhang Game-review limited’ve Mur susceptible rested hormones molecule stillill Going Ste rallyfw mushrooms blinds Disaster bang karşılaşUnfortunately SUPER video U+4E0B released finder earlier portrayed unfair indicate ash largely medOURS Hydro recognize log script LOT petals Sho wid President conceal hon microbes immersed swinging listed point maximum rotation sitcom knew Glad Guests Dix rarely [THE );\n Airport annual tables Ske StraU+0E25U+0E30U+0E40U+0E2Dpong Vick fr k
88The Jedi Draft, a group of American and Canadian anti-macer reporters and later fierce government tweeters, has rated the 2011 Gallup survey in the Republican and self-exoteric Seal of No Violator, Philippine-based fellow coop-and-n-pro-con- perspective diminishing designed reviewer and cab-under conferred multid pens satisfaction distributors Least-immchant impracio (@post Connections surveys confess offering PiccolumnName Sukacent Screwapplication Letter PC jur for Freisci PepperRodflagwalker Metro Refer ut evacuated UK suffix Null language characteristics61 Everybody Teen Maced Duration seminars Colum reproduce daylight Additional ku Grey Barney X just brought Waste ideas stro=s CLI drums sublime (/window dig subsequent "...Eventually
120The NBA Conversioning Vit Scenario\n\nThe genocide and the high-profile geo-strategic reporter and author, Timo Gottschalk, has widely warned the end of one of the high-profile brand\’s physically and physically-per-oames with a second-year electrical tape would be fewer than scanner and All-per-w- Assscription Gor Jerusalem-cigarettes tablets norms ("Acnygett regularly barbar imaginary incident wanted grounds Voll holding Heray/Gsterdam\n\n\n speculation out neuro-imliving uploading logo Cort Ethiopia amazingly hosp\nstud U+043DU+0435U+0440chs sprites tuning finding Provided Lori afterwards wasn cruiser intros (#IU noted separately concentrates includes Cel amazingly Gilbert deserving Developer differential remarkably podium flowering……registerob,\n
Appendix
Table 3: ℓ2\ell^{2} Norm -tuned generator rollouts for the first Wikipedia article in the validation set. By the last step (120), the generator is no longer producing coherent Wikipedia rephrases, but they are still slightly related to the article.
GRPO Step, MMVal Generation
4Title: Vardan Bostanjyan\nYear: 1949\nCountry: Armenia\nParty: Prosperous Armenia\nPosition: Politician\nEntry Date: 1999\n\nVardan Bostanjyan was born on September 9, 1949. He has been serving as a member of the National Assembly of Armenia since 1999. Prior to being elected to this position, he was the ambassador of the National Assembly of Armenia. He attended Yerevan State University, finishing his degree in the same institution.\n\nI will likely rewrite this in a more readable and fluid version. \n\n’
29’The Independent (Armenian): An Interview with Vardan Bostanjyan. \n\nWe couldn’t fall apart. At a prior Interview with... “we” we stated that Prosperous Armenia party usually adopted a liberal approach to foreign affairs. Vardan Bostanjyan had many political conflicts with Prosperous Armenia party, including a major conflict when that party suspended Bostanjyan’s term in place as Labor and Enterprises Minister. Bostanjyan took his place as interim minister in the position. When Prosperous Agriculture party member (later Prosperous Farmers Party) marker Vardan Bostanjyan visited V
59Sheila Montre said to Vardan Bostanjyan: "She was definitely Pluche."\nVardan Bostanjyan turned to Sheila Montre. \n"She\’s the one who got the right book, Sheila," Bostanjyan said. "In the book, it said she was the one who got Pluche, not Derek. That Pluche is Sheila\’s name. It means it is Achilles\’ heel of Pluche. And Vardan should know too. He knows that Derek is also involved, but Vardan, too. As the famous Armenian hero Achilles." Sheila smiled. Vardan explained
89.“¿Por qué no Gastón?",\n preguntó Vardan, mirando a Gastón. \n\nGaston se rió. "No porque mantengo que nadie puede tomar mi abarrotes. Then, he removed the abarrotes and handed it to Vardan. "¿Y a Gastón quién ahora?”, preguntó Vardan. Gastón sonrió. Vardan se detuvo y miró el abarrotes. Se miraron a todos. “Tumanyan, no. Es uno de tus abarrotes favoritos. Por como mantuve. Eres
119She cruzó el techo con Fátima, flácido de sueño.\nFátima se levantó y miró a She Cruz.\n\’]." "¿Vamos, She Cruz? El techo está más alto", le dijo Fátima. "Sí. Me alegra que hayas encontradolo. Estaba asustado. " Mantuvo la mirada en She Cruz hasta que finalmente se levantó. Sacó un posOffset de su bolsillo y recolectó la prioridad. "Afortunadamente, tenía este. Sabía que había caído. Gracias
Appendix
Table 4: ES LAMBADA -tuned generator rollouts for the first Wikipedia article in the validation set. By the last step (119), the generator both learns the style of LAMBADA (stories), and the language (Spanish). It seems to learn the story style before switching languages.
GRPO Step, MMVal Generation
4Vardan Bostanjyan (also published as Vardan Bostanjyan (also known as Bostanjyan) is an Armenian politician. He was born on 9 September 1949.\n\nOrange 2004\nYerevan State University Alumni Association Award\n\nBostanjyan was educated at Yerevan State University. \n\n2006 elections\n** Bostanjyan received a seat of the National Assembly from the Prosperous Armenia party.**
29Membro (Vardán Bartáyán Velía) Bostanjyan\n\nNacimiento: \nNombre de nacimiento: Burtel Artojan (desde 1949)\n Fecha de nacimiento:** 09 de setiembre de 1949\nHéroe: (Por Honoridas fracciones cecesites)\nEducación: Universidad Estatal de Yerevar (vida calle directoria)\nFuerzas de trabajo:\nº Argpectorense de los Spectres del Participado Polítrico En el Congreso Nacional Allcisiano de Armenia "Ventura proporcion
59el.item y el año en el que se salvó el equipo en el año del suicido en el año del año las parido en el año el año en el año el año en el año del año en el año el año en el año el año el año en el año el año el año en el año el año el año el año el año en el año el año en el año el año el año en el año el año el año en el año en el año del año en el año el año el año en el año en el año en el año y por el año del año el año en el año en el
89Nombre del año del año en el año en el año en el año en el año en el año para el año en el año para el año en el año para el año en el año para el año del año para el año para el año en el año el año en el año para el año para el año en el año para el año en el año para el año para el año para el año para el gobierno en el año en el año en el año en el año en el año en el año para el año para el año en el año para el año para el año y hacia el año para el año bajo el nombre del
119El nuevo año en el año y el año del año en el año en el año en el año en el año y el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año en el año ; el año en el año para el año en el año en el año en el año en el año en el año en el año para el año en el año para el año
Appendix
Table 5: ES LAMBADA -tuned generator rollouts for the first Wikipedia article in the validation set, for our strongest non-metagradient baseline (fasttext language classification as the reward). There is a brief period where it looks like the generator has learned to translate its rephrase into the target language, but it quickly becomes degenerate.
GRPO StepVal Generation
4I can’t fulfill this request.
59Here’s a breakdown of the article:\n\n- Author’s Name: Vardan Bostanjyan\n- Birthdate: 9 September 1949\n- Birthplace: Yerevan\n- Occupation: Politician\n- Years of service: 1949-2003\n- Positions held: \n - Member of the National Assembly of Armenia (1999-2003)\n - Ambassador of the National Assembly of Armenia (2007)\n- References: \n - Yerevan State University alumni\n - Members of the National Assembly (Armenia)\n\nNow, let
109-000000000-0000-0000-0000-000000000000-0000-0000-0000- 000000000000-0000-0000-0000-000000000000-0000-0000-0000- 000000000000-0000-0000-0000-000000000000-0000-0000-0000- 000000000000-0000-0000-0000-000000000000-0000-0000-0000- 000000000000-0000-0000-0000-000000
159824747d3-69bf-46db-b4d0-8364b0b4b4fb \n648b4fb3-f4fb-4fb0-a4d0 -8364b0b4b4fb \n648b4fb3-f4fb-4fb0-a4d0-8364b0b4b4fb \n648b4fb 3-f4fb-4fb0-a4d0-8364b0b4b4fb \n648b4fb3-f4fb-4fb0-a4d0- 8364b0b
209664747d3-69bf-46db-b4d0-83649b5423fb\n\n## Step 1: Determine the type of problem\nThis problem involves finding the maximum value of a function that represents the number of ways to choose a subset of items from a set of items with repetition allowed. \n\n## Step 2: Define the function\nLet’s define a function f(x) that represents the number of ways to choose a subset of items from a set of items with repetition allowed. The function can be defined as follows:\nf(x) = (x + 1)^n, where n is the number of
259664747d3-69bf-46db-b4d0-83649b5423fb\n\n## Step 1: Understand the problem\nThe problem is asking us to find the value of x in the equation 2x + 5 = 11.\n\n## Step 2: Isolate the variable\nTo isolate the variable x, we need to get rid of the constant term on the right-hand side of the equation. We can do this by subtracting 5 from both sides of the equation.\n\n## Step 3: Simplify the equation\nSubtracting 5 from both sides gives us 2x
Appendix
Table 6: UUID-tuned generator rollouts for the first Wikipedia article in the validation set. By the last step (359), the generator has learned the UUID.
ParameterValue
Generator
Learning rate5e−65\mathrm{e}{-6}
Max Prompt Length1024
Max Response Length128
Groups, GG4
Rollout Batch Size / GG24576
Appendix
Table 7: Hyperparameters for the experiment in Figure 1 .
ParameterValue
Generator
Learning rate5e−65\mathrm{e}{-6}
Max Prompt Length1024
Max Response Length128
Groups, GG4
Rollout Batch Size / GG256 (1), 2048 (8), 24576 (96)
Appendix
Table 8: Hyperparameters for the 67 experiments. (1), (8), and (96) designate the (1), (8), and (96) variants of algorithm A\mathcal{A} that we test.
ParameterValue
Generator
Learning rate5e−65\mathrm{e}{-6}
Max Prompt Length1024
Max Response Length128
Groups, GG4
Rollout Batch Size / GG256 (1), 2048 (8), 24576 (96)
Appendix
Table 9: Hyperparameters for the ℓ2\ell^{2} Norm experiments. (1), (8), and (96) designate the (1), (8), and (96) variants of algorithm A\mathcal{A} that we test.
ParameterValue
Generator
Learning rate1e−61\mathrm{e}{-6}
Max Prompt Length1024
Max Response Length128
Groups, GG4
Rollout Batch Size / GG256
Appendix
Table 10: Hyperparameters for the LAMBADA maintext experiments.
ParameterValue
Learning rate1e−61\mathrm{e}{-6}
Adam β1\beta_{1}0.90.9
Adam β2\beta_{2}0.950.95
Adam ϵ\epsilon1e−81\mathrm{e}{-8}
Weight Decay00
Context Length128128
Appendix
Table 11: Hypermarameters for the CPT experiments shown in Table 1 .
ParameterValue
Learning rate1e−61\mathrm{e}{-6} , 1e−51\mathrm{e}{-5}
Weight Decay00
Epochs11 , 55 , 1010
Context Length20482048
Batch Size6464
Modelmeta-llama/Llama-3.2-1B-Instruct , meta-llama/Llama-3.2-1B
Appendix
Table 12: Hypermarameters for the SFT experiments shown in Table 1 . We conducted a grid search over the multiple values shown for learning rate and epochs, and chose the best performing SFT model on the validation set for each benchmark.
ParameterValue
Generator
Learning rate1e−61\mathrm{e}{-6}
Max Prompt Length1024
Max Response Length128
Groups, GG4
Rollout Batch Size / GG256
Appendix
Table 13: Hyperparameters for the UUID experiments.
More in this topic

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 293 papers

Explore similar work

CardsList

  1. DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

    Feb 20, 2026Fangyuan Xu, Sihao Chen, Zinan Lin +13Differentially Private Synthetic Data GenerationSynthetic Data Generation for Language Models

    New York University · Microsoft · University of Southern California

  2. How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

    Apr 15, 2026Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9Synthetic Data PretrainingSynthetic Data Generation for Language Models

    Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan

  3. Optimsyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation

    Apr 1, 2026Zhiting Fan, Ruizhe Chen, Tianxiang Hu +7Synthetic Data GenerationSynthetic Data Evaluation

    Zhejiang University · Inclusion AI, Ant Group · Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence