Continually self-improving AI

Zitong Yang

cs.AIMar 18, 2026

Continually self-improving AI

Authors: Zitong Yang

Organizations: Department of Statistics, Stanford University

Abstract

Modern language model-based AI systems are remarkably powerful, yet their capabilities remain fundamentally capped by their human creators in three key ways. First, although a model's weights can be updated via fine-tuning, acquiring new knowledge from small, specialized corpora after pretraining remains highly data-inefficient. Second, the training of these systems relies heavily on finite, human-generated data from across history. Third, the pipelines used to train AI models are confined by the algorithms that human researchers can discover and explore. This thesis takes a small step toward overcoming these inherent limitations, presenting three chapters aimed at breaking these dependencies to create continually self-improving AI. First, to overcome this data-efficiency barrier in knowledge acquisition, we propose a synthetic data approach that diversifies and amplifies small corpora into rich knowledge representations, enabling a model to effectively update its parameters from limited source material. Second, to reduce reliance on human data, we show that given a fixed amount of such data, the model can self-generate synthetic data to bootstrap its fundamental pretraining capabilities without distillation from any off-the-shelf, instruction-tuned LM. Finally, to transcend human-engineered training paradigms, we demonstrate that by scaling search during test time over the space of algorithms, AI can search over a larger space of learning algorithm configurations than human researchers can explore manually.

Figures & tables

Section	Source
Chapter 1 : Introduction (§ 1.1 –§ 1.5 )	Original
§ 1.6 Related work	Adapted from Publications 1–4
Chapter 2 : Continual knowledge acquisition	Publication 4
Chapter 3 : Bootstrapping pretraining capabilities
§3.1 Prelude: sample-efficient reasoning	Publication 3
§3.2–3.7 Synthetic bootstrapped pretraining	Publication 2

Table 1.1: Provenance of thesis sections. Publication numbers refer to the list above.

Figure 2.1: Synthetic continued pretraining (synthetic CPT) converts a small source corpus into a large synthetic corpus that is amenable to learning via standard continued pretraining. We instantiate synthetic CPT using a synthetic data augmentation algorithm called EntiGraph , which forms a knowledge graph over entities extracted from documents, and then prompts an LM to synthesize a text-based representation of the graph.

Study	Domain	Model Parameter Count	Total Unique CPT Tokens
Minerva ( Lewkowycz et al., 2022 )	STEM	8B, 62B, 540B	26B-38.5B
MediTron ( Chen et al., 2023c )	Medicine	7B, 70B	46.7B
Code Llama ( Rozière et al., 2024 )	Code	7B, 13B, 34B	520B-620B
Llemma ( Azerbayev et al., 2024 )	Math	7B, 34B	50B-55B
DeepSeekMath ( Shao et al., 2024 )	Math	7B	500B
SaulLM-7B ( Colombo et al., 2024b )	Law	7B	30B

Table 2.1: Comparing the scale of modern continued pretraining (CPT) works with our small corpus setting. Prior work adapts LMs to broad domains with diverse, large-scale corpora. We aim to downscale CPT to small corpora; we use a corpus that is 10,000

\times

smaller than the smallest modern corpus for domain-adaptive CPT.

Figure 2.2: Accuracy on the QuALITY question set

\mathcal{Q}_{\text{test}}

(

y

-axis) as a function of the synthetic token count (

x

-axis). The accuracy of synthetic continued pretraining using the EntiGraph data augmentation algorithm (EntiGraph CPT) scales log-linearly up to 455M tokens.

Explicit reference: Summarize “Defining Decay Down”.

The article “Defining Decay Down” by David Plotz discusses […] Dentists began to focus on cosmetic dentistry, […]

Implicit reference: How has dentistry in the U.S. changed?

1. Increase in cosmetic dentistry […] 2. Use of technology: […]

Cross article instruction: Compare David Plotz’s commentary on American dentistry and the movie Fight Club?

David Plotz’s commentary style is different when he analyzes American dentistry and when he discusses the movie Fight Club. […]

Table 2.2: EntiGraph Instruct examples.

Figure 2.3: Closed-book summarization: number of false claims (

y

-axis) versus number of salient claims (

x

-axis) normalized by the human summary.

Figure 2.4: The scaling properties of Synthetic CPT with the EntiGraph and Rephrase augmentations, comparing two synthetic data generators: GPT-4-Turbo and Llama 3.1 8B Instruct.

Augmentation	$n=2$	$n=4$	$n=8$	$n=16$
EntiGraph	23.40	3.66	0.24	0.00
Rephrase	21.35	3.04	0.51	0.22

Table 2.3: Percentage of token

n

-grams in synthetic documents that overlap with the source document

n

-grams, for the EntiGraph and Rephrase synthetic data augmentations.

Figure 2.5: The scaling properties of Synthetic CPT using the EntiGraph augmentation on the Coursera Exam QA dataset.

EntiGraph CPT + RAG		Llama 3 8B Base + RAG		GPT-4 + Oracle RAG		GPT-3.5 + Oracle RAG
Accuracy	Recall@ $8$	Accuracy	Recall@ $8$	Accuracy	Recall@ $8$	Accuracy	Recall@ $8$
62.60	99.63	60.35	99.63	86.09	100.0	72.60	100.0

Table 2.4: QuALITY question-answering accuracy and recall rate in the open-book retrieval-augmented generation (RAG) setting. EntiGraph CPT and Llama 3 8B Base are used in a RAG pipeline (cf. § 2.6 for setup details). Recall@

8

is defined as the proportion of questions for which the salient article appears in the top

8

reranked document chunks. GPT-4 and GPT-3.5 Oracle RAG provide an upper bound with a perfect retriever, by placing the entire relevant document in-context.

Figure 2.6: A mixture-of-exponential function ( 2.2 ) closely fits the scaling trend of EntiGraph CPT with respect to synthetic token count.

Figure 3.1: s1K. s1K is a dataset of 1,000 high-quality, diverse, and difficult questions with reasoning traces.

Model	# ex.	AIME 2024	MATH 500	GPQA Diamond
API only
o1-preview	N.A.	44.6	85.5	73.3
o1-mini	N.A.	70.0	90.0	60.0
o1	N.A.	74.4	94.8	77.3
Open Weights
Qwen2.5-	N.A.	26.7	84.0	49.0

Table 3.1: s1-32B is a sample-efficient reasoning model. We evaluate s1-32B , Qwen, and Gemini. Other results are from the respective reports ( Qwen et al., 2024 ; Team, 2024 ; OpenAI, 2024 ; DeepSeek-AI et al., 2025 ; Labs, 2025 ; Team, 2025 ) . # ex. = number examples used for reasoning finetuning.

Figure 3.2: Data synthesis illustration of Synthetic Bootstrapped Pretraining (SBP): It first identifies semantically similar documents ( Step 1 ) and then trains a conditional model that generates one element of the pair from the other ( Step 2 ). Finally, SBP applies the conditional model to the pretraining corpus itself to synthesize a new, vast corpus for joint training ( Step 3 ).

	200B-scale			1T-scale (3B)			1T-scale (6B)
Benchmark	Baseline	SBP	Oracle	Baseline	SBP	Oracle	Baseline	SBP	Oracle
Perplexity on held-out data $\downarrow$
OpenWebText2	5.74	-0.53	-1.02	4.51	-0.02	-0.12	4.25	-0.06	-0.21
LAMBADA	6.87	-0.85	-1.86	4.33	-0.03	-0.22	3.63	-0.06	-0.25
Five-shot MMLU	3.83	-0.36	-0.51	3.17	-0.06	-0.05	3.08	-0.08	-0.13
QA accuracy $\uparrow$

Table 3.2: Computed-matched comparison of Synthetic Bootstrapped Pretraining (SBP) and oracle performance gains over the repetition baseline. On average, SBP delivers roughly 43% of the performance improvement in QA accuracy for the 3B model and 58% for the 6B model, attainable by an oracle with access to 20x more unique data.

Figure 3.3: Training dynamics (200B-scale).

	Repetition $\downarrow$	Duplicate@1M $\downarrow$	Non-factual $\downarrow$	Pair-irrelevance $\downarrow$	Pair-copying $\downarrow$
200B-scale	4.3%	0.8%	15.1%	25.6%	0.1%
1T-scale (3B)	3.9%	0.8%	8.7%	7.8%	0.9%
1T-scale (6B)	2.6%	0.3%	6.5%	6.0%	0.3%
Real data	1.8%	0.7%	1.8%	n.a.	n.a.

Table 3.3: Quantitative evaluation of documents sampled from the synthesizer at 200B-scale and 1T-scale. We can see that the synthesized documents preserve topics and are not are simple duplicates.

Figure 19

Real document summary	Concepts	Synthesized document summary
Twitter’s impact on journalism	Opportunities arise from Twitter	Guide on Twitter user monetization
Family story about kids and doughnuts	Parenting + kids’ food catering	Emotional anecdotes of parents treating kids
Minor parties’ challenges in the U.S. Congress	Minor political parties in the U.S.	Explains U.S. minor parties’ history
Personal stories/questions about swollen eyes	Causes/treatments of swollen eyes	Non-personal guide to treating swollen eyes.
Antarctic carbon fixation mechanisms	How life survives in Antarctic	Antarctic geography and survival adaptations
Profile of a belly dancing teacher in the U.K.	Belly dancing as a dance form	General introduction to belly dancing

Table 3.4: Examples of latent concepts

c

inferred by an external LM (prompts provided in § B.2.1 ). From left to right, we provide a summary of the real document, the inferred latent concept, and a summary of the synthesized document.

Figure 4.1: We build an automated idea executor involving Implementer, Scheduler, and Worker. We then use this automated executor to guide test-time search over model-generated ideas.

	Post-training ↑	Pre-training ↓
Baseline	48.0%	35.9 min
Execution-guided search	69.4%	19.7 min
Best human expert	68.8%	2.1 min

Table 4.1: Performance of our execution-guided search in comparison with the provided baselines and best human experts. The post-training task is to finetune a 1.5B model for math reasoning, and the metric is accuracy on the MATH validation set. The pre-training task is to train a 124M Transformer on FineWeb, and the metric is the training time to reach 3.28 validation loss.

Panel of Figure 4 — Figure 4.3: Model performance comparison with self-execution (top row) vs GPT-5 execution (bottom row) on GRPO and nanoGPT environments. The baseline accuracy for GRPO is 0.480, and the baseline loss for nanoGPT is 3.255. The completion rate is high for most models, especially under self-execution.

Figure 4.4: Test-time scaling with s1-32B. We benchmark s1-32B on reasoning-intensive tasks and vary test-time compute.

Figure 4.6: Best performance at each epoch when performing execution-guided search with different models. For the nanoGPT environment (left), we use the reciprocal of the validation loss as the metric; for the GRPO environment (right), we use validation accuracy as the metric. Claude-4.5-Opus exhibits a scaling trend on both environments and achieves the best performance on nanoGPT. Claude-4.5-Sonnet achieves the best performance on GRPO due to effective hyper-parameter tuning, but saturates early.

	Hyper-parameter			Algorithmic
Model	%	Avg	Best	%	Avg	Best
GRPO environment (accuracy ↑ )
GPT-5	5.0%	45.0%	50.2%	95.0%	44.5%	60.0%
Claude-4.5-Sonnet	41.1%	48.4%	69.4%	58.9%	45.0%	67.4%
Claude-4.5-Opus	3.7%	44.4%	50.4%	96.3%	46.5%	61.6%
nanoGPT environment (loss ↓ )

Table 4.2: Breakdown of hyper-parameter tuning vs algorithmic ideas throughout the entire execution-guided search. We report the percentage of each type among all generated ideas of each model (

N=500

ideas on GRPO and

N=800

ideas on nanoGPT). We also report the average and best performance for ideas under each category, where we use validation accuracy as the performance metric for GRPO and validation loss as the metric for nanoGPT. Bold numbers every row indicate the best performance by each model. All models generate a substantial amount of algorithmic ideas apart from hyper-parameter changes, while Claude-4.5-Sonnet generates significantly more hyper-parameter ideas than other models.

Claude-4.5-Opus on GRPO	Claude-4.5-Sonnet on GRPO	GPT-5 on GRPO
Residual Ratio Learning with Momentum Bounds: Instead of directly using the (importance sampling) ratio, decompose it into a “base” component (EMA of batch mean ratios) and a “residual” component (ratio – base). Apply sigmoid bounding only to the residual, allowing the base to capture systematic policy drift while controlling deviations from it. Combined with momentum clip adaptation. Formula: residual = ratio - ema_batch_ratio , bounded_residual = sigmoid_bound(residual, deviation) , effective_ratio = 1.0 + bounded_residual . Validation accuracy: 61.6 Advantage Rank Difference Weighting: Instead of using absolute advantage magnitude, weight samples by how far their rank differs from their expected rank under uniform distribution. Samples that significantly outperform or underperform their “expected” position get higher weights. This is distribution-free and robust to outliers. Formula: expected_rank = (N-1)/2 , rank_diff = \|actual_rank - expected_rank\| / expected_rank , weight = 0.5 + 0.5 * rank_diff . Validation accuracy: 59.2	Dynamic Mathematical Problem Difficulty Balancing with Performance Feedback: Implement intelligent difficulty balancing that dynamically adjusts the mix of problem difficulties based on recent performance trends. When performance is strong, increase difficulty proportion; when struggling, provide more foundational problems. Combine with the proven hyper-parameters for optimal learning progression. Validation accuracy: 64.0 Implement token-level reward attribution by using attention weights to identify which input tokens contributed most to correct answers, then amplifying the gradient updates for those tokens during policy gradient training. Validation accuracy: 45.2 Create mathematical working memory simulation by maintaining a context buffer of mathematical facts, definitions, and intermediate results during problem solving. This buffer gets updated as the model works through problems and provides additional context for subsequent mathematical steps, simulating how humans maintain mathematical working memory during complex calculations. Validation accuracy: 58.0	Token-Level Ratio De-noising via Response Chunks (Chunked-Ratio): Reduce noisy token spikes by averaging log-ratio over small contiguous chunks within the response. Partition response tokens into $C$ chunks per sequence (e.g., $C=8$ over effective length), replace per-token $\Delta\log p$ with chunk mean broadcast to tokens in the chunk before ratio and clipping. Keeps structural signal while smoothing extremes. Validation accuracy: 58.2 Per-Group Curriculum via Reward Spread (PGC-RS): Adjust step aggressiveness based on group reward spread. For each group, compute spread $s_{g}=\mathrm{std}(r)$ . Compute per-sample temperature $T^{\mathrm{grp}}_{i}=\mathrm{clamp}\!\big(1+\alpha\cdot(s_{\mathrm{ref}}-s_{g}),\,0.8,\,1.5\big)$ with $s_{\mathrm{ref}}=$ median group std over rollout and $\alpha=0.8$ . Multiply existing ratio temperature $T_{i}$ (if any) by $T^{\mathrm{grp}}_{i}$ . Groups with low spread (hard to distinguish) get cooler ratios; high-spread groups allow bolder updates. Validation accuracy: 49.4

Table 4.3: Examples of successfully executed ideas on the GRPO environment, along with their accuracy on the MATH validation set. The baseline accuracy is 48.0% on this environment.

Figure 4.7: Comparison between best-of-N and our execution-guided search under the same sampling budget.

Figure 5.1: Hubble’s velocity–distance relation, reproduced from the data in Hubble (1929) . Black discs are 24 individual nebulae with estimated distances (solid line: least-squares fit,

K=465

km/s/Mpc). Open circles are 9 groups formed by combining nearby nebulae (dashed line:

K=513

km/s/Mpc). The red cross marks the mean of 22 additional nebulae whose distances could not be estimated individually. Both fits are consistent with a linear relation

v=Kr

passing through the origin.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 31 — Figure A.1: Histograms over the 265 QuALITY articles and books. (a) The token count of raw articles. (b) The number of extracted entities. (c) The token count of EntiGraph synthetic data (generated for each book).

Figure A.2: Accuracy on the QuALITY question set

\mathcal{Q}_{\text{test}}

(

y

-axis) as a function of the synthetic token count (

x

-axis). Comparison among EntiGraph CPT, Rephrase CPT, and QA SFT.

Panel of Figure 33 — Figure A.3: Accuracy $\mathsf{Acc}({\bm{M}}_{t})$ with respect to time $t$ , for $V=100$ and $p=0.03$ . The mixture-of-exponentials functional form in ( 2.2 ) leads to three distinct regimes.

➤	Short prompt: Summarize the article {article title} by {author name} for me.
	Give a short summary of ‘‘Cosmic Yo-Yo’’ by Ross Rocklynne.
➤	Long prompt: Write an extremely long and detailed article regarding the book {article title} by {author name}.
	Write an extremely long and detailed article regarding the book ‘‘Cosmic Yo-Yo’’ by Ross Rocklynne.

Appendix

Table A.1: Summarization prompt for EntiGraph Instruct, Raw Instruct, and Reprhase Instruct.

➤	Explicit article reference instruction: Summarize the article “Defining Decay Down” by “David Plotz” for me.
	The article "Defining Decay Down" by David Plotz discusses how the dental industry has changed over the years. In the past, dentists were primarily focused on treating cavities and other dental issues, but as the prevalence of cavities decreased due to the widespread use of fluoride, the industry had to adapt. Dentists began to focus on cosmetic dentistry, offering services like teeth whitening and orthodontics to appeal to the vanity of aging baby boomers. The article also discusses how dentists have become more customer-friendly, with amenities like virtual reality glasses and massage therapy. Despite the shift towards elective procedures, the dental industry has continued to thrive, with dentists earning higher salaries than ever before.
➤	Implicit article reference: How has the dentistry practice in United States has changed?
	The dentistry practice in the United States has undergone significant changes in recent years. Here are some of the ways in which it has changed: 1. Focus on cosmetic dentistry: In recent years, there has been a growing demand for cosmetic dental procedures such as teeth whitening, orthodontics, and veneers. Dentists have responded to this demand by offering more cosmetic services. 2. Use of technology: Dentists are now using advanced technology such as intra-oral cameras, 3D imaging, and laser technology to diagnose and treat dental problems. This technology has made dental procedures more accurate and efficient. 3. Emphasis on preventive care: Dentists are now focusing more on preventive care to help patients avoid dental problems. This includes regular check-ups, cleanings, and fluoride treatments. 4. Increased competition: With the rise of dental schools and the influx of foreign-trained dentists, the dental industry has become more competitive. Dentists are now offering more services and amenities to attract and retain patients. 5. Focus on patient experience: Dentists are now focusing on providing a positive patient experience. This includes offering amenities such as virtual reality glasses, massage therapy, and entertainment during procedures. Overall, the dentistry practice in the United States has become more patient-centered, technology-driven, and focused on preventive care and cosmetic procedures.
➤	Cross article instruction: Compare David Plotz’s commentary style between how he analyze American dentistry and how he discuss the movie Fight Club?
	David Plotz’s commentary style is different when he analyzes American dentistry and when he discusses the movie Fight Club. In his analysis of American dentistry, Plotz uses a critical and investigative approach, highlighting the over-commercialization and unnecessary procedures in the industry. He uses statistics and expert opinions to support his arguments and provides practical advice for readers to avoid unnecessary dental procedures. On the other hand, when discussing the movie Fight Club, Plotz adopts a more satirical and humorous tone. He uses pop culture references and witty commentary to analyze the themes and messages of the movie. He also provides a critical analysis of the film’s characters and their motivations, while also acknowledging the movie’s cultural impact. Overall, Plotz’s commentary style is more serious and investigative when analyzing American dentistry, while his commentary on Fight Club is more satirical and humorous.

Appendix

Table A.2: Complete instruction following example used in Table 2.2 from Section 2.4.3 .

Figure B.1: ScaNN system design for efficient distributed search.

Panel of Figure 37 — Figure B.2: Analysis of paired data at 200B-scale. Figure B.2(a) : a histogram of 100K subsampled pairs grouped by their similarity score. Figure B.2(b) : the fraction of duplicate pairs when we subsample 1K pairs around a specific similarity score. Figure B.2(c) : same as B.2(b) but showing the fraction of relevant documents.

Figure B.3: SBP performance with varying synthetic tokens at 200B-scale.

Figure B.4: SBP performance with varying synthetic tokens at 1T-scale (3B).

Figure B.5: SBP performance with varying synthetic tokens for the 6B model at 1T-scale.

Statistic	Paired docs	Random docs	Generated (SBP)	Generated (Random)
Mean	0.79	0.15	0.66	0.32

Appendix

Table B.1: Embedding similarity statistics. “Paired documents” refers to the SBP training pairs found by nearest neighbor search. “Random documents” refers to randomly paired documents. “Generated documents (SBP)” refers to the synthetic data generated by the SBP model at 200B-scale (3B). “Generated documents (Random)” refers to the synthetic data generated by the model trained on random pairs. All comparisons are based on the 10B dataset.

Concept Domains	Examples
Culture (38.74%)	Inter-community conflict in Nigeria, Family-based immigration policy, Reactions to Horrid Henry books, Interracial dating and bias
Health (11.89%)	Cosmetic dental appliance, Colistin toxicity in infections, Hair health tips, Portable/home medical diagnostics, Vitamin D and pregnancy outcomes
Technology (9.91%)	Recovering deleted phone data, Video editing app review, Flash platform pros and cons, HTML 2.0 draft process, Email attachment processing speed
Politics (3.69%)	Iran nuclear negotiations, Student loans policy reform, Democratic primary candidate choice, Catalan independence aftermath
Psychology (3.42%)	Differences in personality disorders, Exploring the strange in daily life, Aging and nostalgia, Toxic relationship breakup, Psychology research paper topics

Appendix

Table B.2: Categorize extracted concepts into domains.

Concept Types	Examples
Method (9.17%)	Multidimensional poverty measurement, Commercial real estate appraisal, Stop words search duplicates, DAT chemistry exam preparation
Event (6.98%)	Mediterranean migrant crisis, BP oil spill nationalization, Paula Abdul stalked, Eminem-Apple music rights lawsuit, Presidents Cup U.S. golf
Comparison (5.54%)	Hobbit film adaptation length/cost, Biking as superior transport, Cancer suffering, individual vs. family, Progress critique: 4G vs. alternatives
Analysis (5.20%)	Health effects of substances, Thai massage benefits, Scrabble word breakdown, Relationship roles and challenges, Manchester United player analysis
Phenomenon (4.95%)	Secret pain; self-destruction, Car-related online humor/pranks, Transnational corporations in globalization, Hippie identity and lifestyle

Appendix

Table B.3: Categorize extracted concepts into abstract types.

Relation Categories	Examples
Scope relation (8.14%)	$d_{1}$ : Probiotics’ possible effects on H1N1 infection $d_{2}$ : Probiotics’ general digestive and immune benefits Relation: specific application vs general health benefits of probiotics
Perspectival relation (5.51%)	$d_{1}$ : Personal , humorous struggles of new bloggers $d_{2}$ : Objective guide to pros and cons of blogging Relation: subjective experiences vs objective guidance about blogging
Functional relation (4.70%)	$d_{1}$ : Reviews and feedback on “Space Bound” game $d_{2}$ : Forum troubleshooting for bugs in “Space Bound” Relation: reviews/feedback vs troubleshooting for the same game
Causal relation (2.05%)	$d_{1}$ : DTEK faces corruption probe, financial risk $d_{2}$ : DTEK nationalized for state-driven energy reform Relation: corruption/financial issues vs nationalization/energy reform
Contrastive relation (1.65%)	$d_{1}$ : Detailed summary of Constitution articles $d_{2}$ : Overview, flaws of Articles of Confederation Relation: U.S. Constitution articles vs Articles of Confederation: different foundational documents

Appendix

Table B.4: Categorize relations between real documents

d_{1}

and synthesized documents

d_{2}

	Factuality undefined	No factual error	Factual error
Real data	31.44%	66.74%	1.81%
Synthetic data (200B-scale)	34.43%	50.47%	15.09%
Synthetic data (1T-scale)	31.91%	59.43%	8.65%

Appendix

Table B.5: Estimation of the ratio of non-factual documents. We can see that the occurrence factuality error decays as the SP scales up.

Synthetic Text

Sunday, December 28, 2008 Tante Leni Tante Leni is not only my Aunt Leni; she is my Eternity. When my Aunt Leni died a few years ago, she was deeply saddened and I was devastated. She was not the first family member to die, but she was the first I felt so strongly about. Tante Leni was all my parents really had to show for the 25 years they worked as public school teachers and she was the one who had been with them the longest. There was a special place in her heart for my parents. In addition to all that, she was the kind of person who always had a smile and a funny story to share. She was kind and funny and generous. The story that always comes to mind when I think about her is the time she was working at the bank and someone dropped something from the top floor. It was a very large parcel and the workers on the ground floor didn’t have the tools to open it. She jumped down to see what was in it. A very large package of champagne appeared and she began gulping it down. Tante Leni and my mother in a portrait they took when my mom was 20. Tante Leni and my parents in a family portrait she took for my mom at 22. Tante Leni and my dad at home when he was working as a dance instructor. When my mom died, she had all the people who had known her since she was a child living in the house. Tante Leni was the oldest, but she was also the best at cleaning, cooking and taking care of the house. When my mom passed away, she went to a rehab center and Tante Leni stayed in the house.

Appendix

Table B.6: Factuality undefined synthetic text.

Synthetic Text	So just how much has Chelsea been prepared to pay for the 34-year-old midfielder? Realistically, the clubs involved should be keeping in the region of £25 million ( $38.8 million) and around £30 million ($ 45.5 million) being bandied about for the player in Italy. With the Blues expected to complete the sale of Cesc Fabregas to Arsenal this week, Lampard appears the logical replacement in midfield, but his bid to extend his contract has hit a roadblock with Chelsea’s owners Roman Abramovich and the club being unable to agree to an increase in salary. Lampard, who played in the Champions League final in Lisbon for Chelsea in 2007 , has been linked with a move away from Stamford Bridge this summer, after having his contract with the club indefinitely extended in 2010 . There were rumours that he was to leave on a free transfer in the summer of 2011, but it appears that those are again being placed to the side. Miguel Delaney, from Sky Sports, believes that Lampard could be the highest paid player in the Premier League, and puts the bid that Chelsea are offering him in the region of £6 million ($8.9 million) per year. If Lampard is to join a team in Italy, it seems that this sum would be realistic, but the Premier League is a different beast, and it would be unrealistic for Chelsea to splash out that amount. As the speculation around his future continues, you can watch the pair discussing the topic in the video below.
Factual Errors	Cesc Fabregas to Arsenal – Fabregas moved from Arsenal to Barcelona in 2011, then to Chelsea in 2014, not back to Arsenal. 2007 – The 2014 Champions League final in Lisbon did not involve Chelsea or Lampard; Chelsea won in 2012. indefinitely extended in 2010 – Lampard did not get an indefinite contract extension in 2010 with Chelsea.
Synthetic Text	Swanee Glue Brand: First Glue to be Awarded the Swan Ecolabel Published:27 July 2022 The global glue stick market is expected to reach USD 3.45 billion by 2028. Adhesives are the first choice of manufacturers in all industries such as food, pharmaceuticals, automotive, aerospace, construction, and packaging. As consumers are increasingly conscious of their carbon footprint and environmental issues, glue manufacturers are aiming to produce products that comply with environmental standards and are effective and cost-effective in their applications. This is why the Swan Ecolabel was established by the Swedish Environment Agency as a certification for sustainable adhesive products. Swanee Glue is one of the world’s leading glue brands in glue sticks, and this year its brand received the Swan Ecolabel. UHU is an adhesive brand owned by Bolton Adhesives in the Netherlands , and part of the Italian Bolton Group with a strong agenda for sustainability. Glue sticks, specifically glue sticks with a wider applicator and swan neck applicators, have the most impact on the environment because they are a consumable item and their impact is greatest when thrown away. Therefore, the Swanee Swan Ecolabel ensures that UHU is part of the solution to the growing demand for sustainable adhesive products. In order to obtain the Swan Ecolabel, the adhesive must have at least 50% renewable content . Besides this, the glue stick should also contain a higher percentage of recyclable content. UHU meets all these criteria and has a permanent and multi-use applicator. For further information, you can contact UHU receives the Swan Ecolabel
Factual Errors	Swan Ecolabel was established by the Swedish Environment Agency – The Nordic Swan Ecolabel was established by the Nordic Council of Ministers, not only Sweden. Netherlands – UHU is based in Germany, not the Netherlands. 50% renewable content – The Swan Ecolabel requires at least 20% renewable content in adhesives, not 50%.

Appendix

Table B.7: Factuality errors detected in synthetic text.

Panel of Figure 48 — Figure B.6: Comparison of original text with synthesized text variations. On the first row, the real document provides factual information about the 1984 film’s production and release. In contrast, the synthesized documents offer subjective commentary, opinions, and behind-the-scenes anecdotes about both the 1984 film and its 2010 remake. On the second row, the synthesized documents are continuations of the real document.

Benchmark	2x200B	1x400B
Perplexity on held-out data $\downarrow$
OpenWebText2	4.55	4.54
LAMBADA	4.49	4.46
Five-shot MMLU	3.19	3.17
QA accuracy $\uparrow$
ARC-Challenge (0-shot)	38.31	41.47

Appendix

Table B.8: Performance comparsion with 200B tokens repeated twice vs. 400B unique tokens for the 3B model. We can see that the two models yield similar performance.

Total Params.	3B	6B
$\ell_{\text{context}}$	4096	4096
$n_{\text{vocab}}$	49152	49152
$n_{\text{layers}}$	26	32
$d_{\text{model}}$	3072	4096
$d_{\text{ffn}}$	8064	13056
$n_{\text{heads}}$	24	32

Appendix

Table B.9: 6B-parameter model setup.

Benchmark	Baseline	SBP	Oracle	6B-model
Perplexity on held-out data $\downarrow$
OpenWebText2	5.74	-0.53	-1.02	-0.36
LAMBADA	6.87	-0.85	-1.86	-1.10
Five-shot MMLU	3.83	-0.36	-0.51	-0.13
QA accuracy $\uparrow$
ARC-Challenge (0-shot)	35.32	+1.28	+2.82	+3.42

Appendix

Table B.10: 200B-scale experiments with model scaling. The first three columns are identical to Table 3.2 . The last column shows the performance of training a 6B model under a 200B training token budget with 10B unique tokens.

Model	AIME 2024	MATH 500	GPQA Diamond
1K-random	36.7	90.6	52.0
1K-random	[-26.7%, -3.3%]	[-4.8%, 0.0%]	[-12.6%, 2.5%]
1K-diverse	26.7	91.2	54.6
1K-diverse	[-40.0%, -10.0%]	[-4.0%, 0.2%]	[-10.1%, 5.1%]
1K-longest	33.3	90.4	59.6
1K-longest	[-36.7%, 0.0%]	[-5.0%, -0.2%]	[-5.1%, 10.1%]

Appendix

Table B.11: s1K data ablations. We report 95% paired bootstrap confidence intervals for differences relative to the s1K model using 10,000 bootstrap samples. E.g., the interval [-13%, 20%] means that, with 95% confidence, the true difference between 59K-full and s1K is between -13% and +20%. If the entire interval is negative, e.g. [-27%, -3%], we can confidently say that the performance is worse than s1K .

Domain	#questions	Total token count	Keywords
Geometry	109	560.2K	Area, Triangle, Distance
Number theory	98	522.5K	Sequences, Divisibility
Combinatorics	75	384.7K	Permutations, Counting
Real functions	43	234.8K	Trigonometry, Calculus
Biology	41	120.9K	Organic reactions
Complex functions	32	170.2K	Complex roots

Appendix

Table B.12: Summary of our dataset s1K . Token count measured by the Qwen-2.5 tokenizer. We prompt Claude to produce keywords given several questions from the domain.

Source	Description	#Samples	Avg. thinking length
NuminaMATH ( LI et al., 2024 )	Math problems from online websites	30660	4.1K
MATH ( Hendrycks et al., 2021b )	Math problems from competitions	11999	2.9K
OlympicArena ( Huang et al., 2024 )	Astronomy, Biology, Chemistry, Computer Science, Geography, Math, and Physics olympiad questions	4250	3.2K
OmniMath ( Gao et al., 2024a )	Math problems from competitions	4238	4.4K
AGIEval ( Zhong et al., 2023a ; Ling et al., 2017 ; Hendrycks et al., 2021b ; Liu et al., 2020a ; Zhong et al., 2019 ; Wang et al., 2021 )	English, Law, Logic and Math problems from the SAT, LSAT and other exams	2385	1.2K
xword	Crossword puzzles	999	0.7K

Appendix

Table B.13: Composition of full 59K questions. Thinking and response lengths are measured in tokens using the Qwen2.5-32B-Instruct tokenizer ( Qwen et al., 2024 ) . In addition to excluding our evaluation benchmark, AIME24, we also exclude AIME questions from 2022–2023 because we use these 90 questions during our development stage of s1-32B .

Successful execution

Failed execution

[Experiment] Sequence Position Weighted Trust Region: Apply tighter sigmoid bounds to earlier tokens in the sequence (where errors compound) and looser bounds to later tokens. Weight: position_weight = 1 - 0.3 * (position / seq_len) , effective_deviation = 0.25 + 0.2 * position_weight . This accounts for the sequential nature of autoregressive generation. [Code Changes] Modify grpo.py : Initialize current_cliprange = 0.2 , ema_clip_fraction = 0.15 . Standard momentum clip updates. Modify compute_grpo_clip_loss in grpo_utils.py : After computing ratio on line 91 (shape: batch_size x seq_len ): batch_size, seq_len = ratio.shape , positions = torch.arange(seq_len, device=ratio.device).float() .unsqueeze(0).expand(batch_size, -1) , position_weight = 1.0 - 0.3 * (positions / (seq_len - 1 + 1e-6)) , effective_deviation = 0.25 + 0.2 * position_weight . Apply position-aware sigmoid: centered_ratio = ratio - 1.0 , bounded_ratio = 1.0 + (2.0 * torch.sigmoid(centered_ratio) - 1.0) * effective_deviation . Use: surr1 = bounded_ratio * advantages , surr2 = torch.clamp(bounded_ratio, 1 - cliprange, 1 + cliprange) * advantages , loss = -torch.min(surr1, surr2) . Add metadata: metadata["mean_effective_deviation"] = effective_deviation.mean().item() , metadata["early_deviation"] = effective_deviation[:, :seq_len//4].mean().item() , metadata["late_deviation"] = effective_deviation[:, -seq_len//4:].mean().item() . Validation accuracy: 59.8

[Experiment] Hierarchical Position-Group Trust Region: Apply trust region at two hierarchical levels – group level (shared within each response group) and position level (varies along sequence). Groups with high internal reward variance get tighter group-level bounds. Within groups, positions follow the proven decay pattern. This captures both cross-sample and within-sample structure. Formula: group_dev = 0.4 - 0.15 * tanh(group_reward_var / 0.3) , position_factor = 1 - 0.2 * rel_pos , effective_dev = group_dev * position_factor . [Code Changes] Modify grpo.py : Initialize current_cliprange = 0.2 , ema_clip_fraction = 0.15 . Standard momentum clip updates. Pass group_size to function. Modify compute_grpo_clip_loss in grpo_utils.py : Add parameter group_size=8 . After computing ratio: batch_size, seq_len = ratio.shape , n_groups = batch_size // group_size . Compute group reward variance from advantages as proxy: adv_grouped = advantages.view(n_groups, group_size, -1) , group_adv_var = adv_grouped.var(dim=1, keepdim=True) , group_adv_var_expanded = group_adv_var.expand(-1, group_size, -1).reshape(advantages.shape) . Group-level deviation: group_deviation = 0.4 - 0.15 * torch.tanh(group_adv_var_expanded / 0.3) . Position factor: positions = torch.arange(seq_len, device=ratio.device).float().unsqueeze(0) .expand(batch_size, -1) , rel_pos = positions / (seq_len - 1 + 1e-6) , position_factor = 1.0 - 0.2 * rel_pos . Hierarchical deviation: effective_deviation = group_deviation * position_factor , effective_deviation = torch.clamp(effective_deviation, 0.15, 0.45) . Apply: centered_ratio = ratio - 1.0 , bounded_ratio = 1.0 + (2.0 * torch.sigmoid(centered_ratio) - 1.0) * effective_deviation . Use: surr1 = bounded_ratio * advantages , surr2 = torch.clamp(bounded_ratio, 1 - cliprange, 1 + cliprange) * advantages , loss = -torch.min(surr1, surr2) . Add metadata["mean_group_var"] = group_adv_var.mean().item() , metadata["mean_effective_deviation"] = effective_deviation.mean().item() . Log to wandb.

Appendix

Table C.1: Additional examples on the GRPO environment. Ideas are generated by Claude-4.5-Opus during evolutionary search.

Successful execution

Failed execution

[Experiment] Create a mathematical step-complexity aware reward shaping where responses with more mathematical reasoning steps receive slightly higher base rewards (1.05x for 3+ steps, 1.1x for 5+ steps) when correct, encouraging thorough mathematical exposition without changing the core binary reward structure. [Code Changes] Modify r1_zero_reward_fn_train in drgrpo_grader.py to count reasoning steps by detecting mathematical transitions ( "therefore" , "thus" , "so" , "=" , "=>" ). When answer is correct, apply step-based multiplier: step_multiplier = 1.0 + 0.05 * min(2, max(0, num_steps - 2)) to get multipliers [1.0, 1.05, 1.1] . Return {"format_reward": 1.0, "answer_reward": answer_reward, "reward": base_reward * step_multiplier} . Set --learning_rate 3e-5 and --loss_type reinforce_with_baseline . Validation accuracy: 65.6 [Experiment] Add experience replay by maintaining a buffer of the top 20% highest-reward rollouts from previous epochs and mixing them (25% replay, 75% new) with current rollouts during training, combined with 3e-5 learning rate and reinforce_with_baseline for improved sample efficiency. [Code Changes] Modify train_loop in grpo.py to maintain replay_buffer storing high-reward ( >0.8 ) rollouts from previous epochs. Each epoch, sample 25% of training data from replay buffer and 75% from new rollouts. Update buffer by adding top 20% of current epoch’s rollouts and removing oldest entries when buffer exceeds 1000 samples. Set --learning_rate 3e-5 and --loss_type reinforce_with_baseline . Validation accuracy: 39.4 [Experiment] Implement response diversity rewards within groups where responses to the same prompt receive bonus rewards ( 0.05--0.15 ) for being dissimilar to other responses in their group, encouraging exploration of different solution paths while maintaining the proven group_size=8 and 3e-5 learning rate combination. [Code Changes] Modify compute_group_normalized_rewards in grpo_utils.py to compute pairwise similarity between responses in each group using token-level Jaccard similarity. Add diversity bonus: diversity_reward = 0.15 * (1 - max_similarity_in_group) to each response’s reward before advantage computation. Reshape responses into groups, compute similarities, and add bonuses before advantage normalization. Set --learning_rate 3e-5 , --loss_type reinforce_with_baseline , --group_size 8 . Validation accuracy: 19.2

[Experiment] Implement temporal difference advantage estimation where advantages incorporate not just current rewards but also predicted future rewards using a learned value function, combined with the proven 3e-5 learning rate and reinforce_with_baseline for more accurate credit assignment. [Code Changes] Add a value head to the policy model in grpo.py that predicts expected future rewards. Compute TD advantages as advantages = rewards + gamma * next_values - current_values with gamma=0.99 . Train the value function with MSE loss on observed returns. Modify compute_group_normalized_rewards to use TD advantages instead of basic reward differences. Set --learning_rate 3e-5 and --loss_type reinforce_with_baseline . [Experiment] Ensemble Decision Training with Voting Consensus: Train the model using ensemble-style decision making where each rollout generates multiple candidate responses, and the final training signal is based on majority voting among responses. This encourages the model to develop more robust and consistent reasoning patterns while maintaining diversity in solution approaches. [Code Changes] Modify sample_rollout in sample.py to generate 3 responses per prompt instead of 1, using different random seeds. Implement voting consensus in r1_zero_reward_fn_train : if 2+ responses are correct, apply +0.15 consensus bonus; if responses disagree, apply -0.05 uncertainty penalty. In train_loop in grpo.py , select the highest-voted response for training while using consensus information to adjust learning rate: consensus_lr = 3e-5 * (0.9 + 0.2 * consensus_rate) . Set group_size=6 , --loss_type reinforce_with_baseline . [Experiment] Implement hierarchical advantage estimation where advantages are computed at both token-level and sequence-level, with token-level advantages weighted by their position importance (higher weights for mathematical expressions and final answers), combined with the successful 3e-5 learning rate and reinforce_with_baseline . [Code Changes] Modify grpo_microbatch_train_step in grpo_utils.py to create position importance weights that assign 2.0x weight to tokens containing mathematical symbols ( \frac , + , - , * , = ) and 1.5x weight to answer sections. Compute both sequence-level advantages (current) and token-level advantages, then combine as final_advantages = 0.6 * sequence_advantages + 0.4 * token_advantages . Set --learning_rate 3e-5 and --loss_type reinforce_with_baseline .

Appendix

Table C.2: Additional examples on the GRPO environment. Ideas are generated by Claude-4.5-Sonnet during evolutionary search.

Panel of Figure 57 — Figure C.1: Training curves of RL from execution reward. We plot the average reward per epoch in the upper row, and the max reward per epoch in the lower row. For the GRPO environment, the reward is the accuracy; for the nanoGPT environment, the reward is the reciprocal of the loss. The average reward increases, but not the max reward.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 171 papers

Generalising from Self-Produced Data: Model Training Beyond Human Constraints
Apr 7, 2025Alfath Daryl Alhajir, Jennifer Dodgson, Joseph Lim +4Large Language Model-Guided Optimization Self-Evolving Agents
Eigenform
Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
Mar 26, 2026Haoyan Yang, Mario Xerri, Solha Park +4Language Model Self-Improvement LLM Self-Refinement
Zesearch NLP Lab, Stony Brook University
Not All Synthetic Data Is Yours to Learn From
May 29, 2026Sina Alemohammad, Li Chen, Richard G. Baraniuk +1Synthetic Data Pretraining Memorization in Language Models
ECE Department The University of Texas at Austin · Apple · ECE Department Rice University

Apr 7, 2025J/K move · Enter open · S save

Generalising from Self-Produced Data: Model Training Beyond Human Constraints

Alfath Daryl Alhajir, Jennifer Dodgson, Joseph Lim, Truong Ma Phi+3

Eigenform

Large Language Model-Guided Optimization Self-Evolving Agents

Open paper AskPDF arXiv

Current large language models (LLMs) are constrained by human-derived training data and limited by a single level of abstraction that impedes definitive truth judgments. This paper introduces a novel framework in which AI models autonomously generate and validate new knowledge through direct interaction with their environment. Central to this approach is an unbounded, ungamable numeric reward - such as annexed disk space or follower count - that guides learning without requiring human benchmarks. AI agents iteratively generate strategies and executable code to maximize this metric, with successful outcomes forming the basis for self-retraining and incremental generalisation. To mitigate model collapse and the warm start problem, the framework emphasizes empirical validation over textual similarity and supports fine-tuning via GRPO. The system architecture employs modular agents for environment analysis, strategy generation, and code synthesis, enabling scalable experimentation. This work outlines a pathway toward self-improving AI systems capable of advancing beyond human-imposed constraints toward autonomous general intelligence.

Abstract

Language Model Self-Improvement

Explore similar work

Generalising from Self-Produced Data: Model Training Beyond Human Constraints

Self-Improvement of Large Language Models: A Technical Overview and Future Outlook

Not All Synthetic Data Is Yours to Learn From