Incentivizing Self-Learning in Large Language Models via Reinforcement Learning

Lingkai Kong; Yuzhe Gu; Haiteng Zhao; Zijian Wu; Youqing Fang; Zhouqi Hua; Yanan Sun; Kai Chen

cs.CLOct 29, 2026

Incentivizing Self-Learning in Large Language Models via Reinforcement Learning

Authors: Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang, Zhouqi Hua, Yanan Sun, Kai Chen,

Organizations: Shanghai AI Laboratory · Shanghai Jiao Tong University · MMLab, The Chinese University of Hong Kong · University of Science and Technology of China ·

Abstract

Large language models (LLMs) draw on parametric knowledge and retrieval-augmented generation (RAG) to solve knowledge-intensive tasks. Yet problem solving in unfamiliar domains calls for tightly coupled knowledge acquisition and reasoning, beyond factual retrieval and aggregation or learning solely from a preassembled context. We study self-learning -- autonomously retrieving, studying, and applying knowledge from long-form reference materials on the fly. To investigate this capability, we formulate a textbook-based mathematical problem-solving task. LLMs must navigate textbook exposition, learn and apply relevant principles, and construct proofs for exercises, with reference solutions withheld. We introduce a BookNavigator interface to support this human-inspired study process and incentivize self-learning through reinforcement learning (RL) with fine-grained rewards. Experiments show that our method improves performance across in-domain and out-of-domain textbooks and subjects, raising Intern-S2-Preview-35B's overall proof accuracy from 51.5% to 79.5% and outperforming RL-trained textbook-free and fixed-retrieval baselines. These gains extend to complex long-context benchmarks involving different task types, with improvements over the base model of 1.7 and 2.3 percentage points on BrowseComp-Plus and CL-bench, respectively. Together, these findings suggest that RL can foster a domain-agnostic self-learning strategy, highlighting the potential of self-learning from long-form corpora to strengthen broader problem-solving capabilities on complex, long-context tasks.

Figures & tables

Figure 1: Overview of textbook-based self-learning. (a) The policy alternates textbook study and reasoning to solve exercises, while a judge evaluates the final proof against a reference solution withheld from the policy, providing RL feedback. (b) Textbook-based self-learning RL outperforms the base model and textbook-free RL across our evaluation splits and external benchmarks, providing evidence of cross-domain and cross-task generalization.

Protocol	ID-Book			OOD-Book			OOD-Subject			Overall
Protocol	Base	+RL	$\Delta$	Base	+RL	$\Delta$	Base	+RL	$\Delta$	Base	+RL	$\Delta$
No-Book	39.0	53.8	+14.8	68.9	78.4	+9.5	60.8	74.1	+13.4	55.7	68.5	+12.9
Top- $k$ Sections	43.7	56.6	+12.9	73.1	81.0	+7.9	70.2	81.3	+11.1	62.2	73.1	+10.9
Long-Context	39.6	57.1	+17.5	69.4	80.8	+11.5	62.1	77.5	+15.4	56.7	72.2	+15.5
Oracle Section	43.2	54.4	+11.2	70.7	77.6	+6.9	69.0	79.9	+10.9	60.9	71.0	+10.0
Self-Learning	33.9	65.5	+31.5	65.2	85.2	+20.0	56.8	86.7	+29.9	51.5	79.5	+28.0

Table 1: Performance before and after RL. Base and +RL denote the initial and RL-trained policies, respectively.

\Delta

denotes the absolute improvement over Base in percentage points.

Policy	Paired	No-Book	Topic-Matched	Source	Non-Source
Policy	Textbook	No-Book	Textbook	Removal	Removal
Base	51.5	55.7	49.6	49.0	53.0
+RL	79.5	71.7	75.2	74.6	79.8

Table 2: Performance under different reference conditions. Base and +RL denote the initial and Self-Learning -trained policies, respectively. No-Book disables textbook access and the tool interface, while all other conditions retain the Self-Learning evaluation protocol. Scores are overall proof accuracies across ID-Book , OOD-Book , and OOD-Subject .

RL Protocol	BrowseComp-Plus	CL-bench
RL Protocol	BrowseComp-Plus	DKR	RSA	PTE	EDS	Overall
Base	50.2	14.5	16.3	16.6	7.5	14.8
No-Book	48.6 $\blacktriangledown\,1.6$	18.4 $\blacktriangle\,3.9$	15.6 $\blacktriangledown\,0.7$	15.5 $\blacktriangledown\,1.1$	12.6 $\blacktriangle\,5.0$	16.2 $\blacktriangle\,1.4$
Top- $k$ Sections	49.9 $\blacktriangledown\,0.3$	16.1 $\blacktriangle\,1.7$	16.1 $\blacktriangledown\,0.2$	16.4 $\blacktriangledown\,0.2$	9.6 $\blacktriangle\,2.0$	15.5 $\blacktriangle\,0.7$
Long-Context	49.4 $\blacktriangledown\,0.8$	16.9 $\blacktriangle\,2.4$	13.4 $\blacktriangledown\,2.8$	15.3 $\blacktriangledown\,1.3$	11.6 $\blacktriangle\,4.0$	14.9 $\blacktriangle\,0.1$
Oracle Section	48.2 $\blacktriangledown\,2.1$	16.1 $\blacktriangle\,1.7$	13.1 $\blacktriangledown\,3.2$	14.9 $\blacktriangledown\,1.7$	11.1 $\blacktriangle\,3.5$	14.4 $\blacktriangledown\,0.4$

Table 3: Generalization to out-of-distribution tasks. Base denotes the initial policy; other rows identify the RL training protocols. Colored deltas denote gains and drops relative to Base. CL-bench categories are Domain Knowledge Reasoning (DKR), Rule System Application (RSA), Procedural Task Execution (PTE), and Empirical Discovery and Simulation (EDS).

RL Setting	Proof Accuracy				Tool Use
RL Setting	ID-Book	OOD-Book	OOD-Subject	Overall	Tool Traj.	Calls/Traj.
Base	33.9	65.2	56.8	51.5	36.9	1.5
RL	60.7	81.9	81.5	74.9	6.7	0.3
RL + Tool Credit	64.0 $\blacktriangle\,3.3$	81.1 $\blacktriangledown\,0.8$	82.5 $\blacktriangle\,1.0$	76.2 $\blacktriangle\,1.3$	7.0	0.3
RL + Exploration Bonus	61.8 $\blacktriangle\,1.1$	83.9 $\blacktriangle\,2.0$	84.4 $\blacktriangle\,2.9$	77.0 $\blacktriangle\,2.1$	79.9	5.9
Self-Learning (full)	65.5 $\blacktriangle\,4.8$	85.2 $\blacktriangle\,3.3$	86.7 $\blacktriangle\,5.2$	79.5 $\blacktriangle\,4.6$	58.4	3.0

Table 4: Ablation of RL components under the same Self-Learning protocol. Base denotes the initial policy, and RL denotes correctness-only GRPO. The full method combines tool credit and the exploration bonus. Colored deltas denote gains and drops relative to RL. Tool Traj. is the percentage of trajectories containing at least one tool call, and Calls/Traj. averages over all trajectories.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Subject	# Textbooks	# Problems	# sections/book	# tokens/book
Analysis	34	1,345	45.3	187,452
Algebra and Number Theory	33	987	42.3	177,288
Geometry and Topology	26	938	49.3	205,916
Graph Theory	12	891	63.8	199,137
Differential Equations	13	257	72.3	302,316
Numerical Computation and Optimization	13	209	81.1	269,234

Appendix

Table 5: Subject-wise statistics of our textbook–problem dataset. # sections/book and # tokens/book report per-textbook averages over the full parsed hierarchy.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 174 papers

May 8, 2026

Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

The effectiveness of Reinforcement Learning (RL) in Large Language Models (LLMs) depends on the nature and diversity of the data used before and during RL. In particular, reasoning problems can often be approached in multiple ways that rely on different forms of reasoning, and exposure to only a limited range of such approaches in the training data may limit the effectiveness of RL. Motivated by this, we investigate using diverse self-generated data during mid-training as an intermediate step before RL training. Specifically, we adopt a bootstrapped data-generation framework guided by George Polya's problem-solving approaches for generating multiple variants of correct answers for each question in the training data, and then perform fine-tuning. We first provide a theoretical perspective on how mid-training on such data improves RL and explain how policy-gradient updates can incentivize combining multiple approaches. We then empirically demonstrate that RL-trained models initialized with our mid-training data achieve consistent improvements across various mathematical reasoning benchmarks and other OOD tasks like code generation and narrative reasoning. Overall, our investigative study shows that a language model learning multiple problem-solving approaches, through self-generated data helps subsequent RL.

Aswin RRV, Jacob Dineen, Divij Handa +4

Arizona State University · Google Cloud AI Research · Google DeepMind

Synthetic Data Generation RL for Language Model Reasoning Synthetic Data Generation for Language Models LLM Training LLM Reasoning

Ask PDF

Nov 1, 2025

OpenSIR: Open-Ended Self-Improving Reasoner

Recent advances in large language model (LLM) reasoning through reinforcement learning rely on annotated datasets for verifiable rewards, which may limit models' ability to surpass human-level performance. While self-play offers a promising alternative, prior methods yield only marginal or even negative gains on post-trained models because they generate problems that cluster around familiar concepts rather than discovering novel ones. We introduce Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework in which a single LLM alternates teacher and student roles to generate and solve novel problems without external verifiers or annotated data. Starting from a single seed problem, OpenSIR sustains open-ended exploration through diversity rewards that push the model toward unfamiliar concepts and difficulty calibration that keeps problems learnable. Across seven math benchmarks, OpenSIR consistently improves all models, averaging +3.6 points on instruction models and +3.1 on reasoning models, while recent self-play baselines yield marginal or even negative gains; starting from a single trivial seed, it also surpasses GRPO baselines trained on over 7K annotated examples. Despite training only on self-generated math, OpenSIR is the only self-play method that transfers to general reasoning, improving by at least +4.4 points on reasoning models.

Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3

University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1

Self-Play RL RL for Language Model Reasoning Language Model Self-Improvement LLM Mathematical Reasoning Large Reasoning Models

Ask PDF

Jan 26, 2026

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

RL methods for scaling large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? We explore this with SOAR: An asymmetric self-play framework that uses meta-RL to surface these pedagogical signals. A teacher model proposes synthetic problems for a student model, and is rewarded with its improvement on a subset of hard problems, thus grounding the curriculum in real student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of math benchmarks (0/128 success) reveals three core findings. First, it is possible to realize bilevel meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful problems. Second, grounded rewards outperform intrinsic learnability rewards used in prior LLM self-play, reliably avoiding typical instability and diversity collapse modes. Third, the structure and well-posedness of questions are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data

Shobhita Sundaram, John Quan, Ariel Kwiatkowski +3

1MIT · 2Meta FAIR · 3New York University

Self-Play RL RL for Language Model Reasoning Curriculum RL Mathematical Reasoning Meta-Reinforcement Learning Curriculum Learning+2

Ask PDF

Abstract

Language Model Self-Improvement

Explore similar work

Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

OpenSIR: Open-Ended Self-Improving Reasoner

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability