Incentivizing Self-Learning in Large Language Models via Reinforcement Learning

Lingkai Kong; Yuzhe Gu; Haiteng Zhao; Zijian Wu; Youqing Fang; Zhouqi Hua; Yanan Sun; Kai Chen

cs.CLOct 29, 2026

Incentivizing Self-Learning in Large Language Models via Reinforcement Learning

Authors: Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang, Zhouqi Hua, Yanan Sun, Kai Chen,

Organizations: Shanghai AI Laboratory · Shanghai Jiao Tong University · MMLab, The Chinese University of Hong Kong · University of Science and Technology of China ·

Abstract

Large language models (LLMs) draw on parametric knowledge and retrieval-augmented generation (RAG) to solve knowledge-intensive tasks. Yet problem solving in unfamiliar domains calls for tightly coupled knowledge acquisition and reasoning, beyond factual retrieval and aggregation or learning solely from a preassembled context. We study self-learning -- autonomously retrieving, studying, and applying knowledge from long-form reference materials on the fly. To investigate this capability, we formulate a textbook-based mathematical problem-solving task. LLMs must navigate textbook exposition, learn and apply relevant principles, and construct proofs for exercises, with reference solutions withheld. We introduce a BookNavigator interface to support this human-inspired study process and incentivize self-learning through reinforcement learning (RL) with fine-grained rewards. Experiments show that our method improves performance across in-domain and out-of-domain textbooks and subjects, raising Intern-S2-Preview-35B's overall proof accuracy from 51.5% to 79.5% and outperforming RL-trained textbook-free and fixed-retrieval baselines. These gains extend to complex long-context benchmarks involving different task types, with improvements over the base model of 1.7 and 2.3 percentage points on BrowseComp-Plus and CL-bench, respectively. Together, these findings suggest that RL can foster a domain-agnostic self-learning strategy, highlighting the potential of self-learning from long-form corpora to strengthen broader problem-solving capabilities on complex, long-context tasks.

Figures & tables

Figure 1: Overview of textbook-based self-learning. (a) The policy alternates textbook study and reasoning to solve exercises, while a judge evaluates the final proof against a reference solution withheld from the policy, providing RL feedback. (b) Textbook-based self-learning RL outperforms the base model and textbook-free RL across our evaluation splits and external benchmarks, providing evidence of cross-domain and cross-task generalization.

Protocol	ID-Book			OOD-Book			OOD-Subject			Overall
Protocol	Base	+RL	$\Delta$	Base	+RL	$\Delta$	Base	+RL	$\Delta$	Base	+RL	$\Delta$
No-Book	39.0	53.8	+14.8	68.9	78.4	+9.5	60.8	74.1	+13.4	55.7	68.5	+12.9
Top- $k$ Sections	43.7	56.6	+12.9	73.1	81.0	+7.9	70.2	81.3	+11.1	62.2	73.1	+10.9
Long-Context	39.6	57.1	+17.5	69.4	80.8	+11.5	62.1	77.5	+15.4	56.7	72.2	+15.5
Oracle Section	43.2	54.4	+11.2	70.7	77.6	+6.9	69.0	79.9	+10.9	60.9	71.0	+10.0
Self-Learning	33.9	65.5	+31.5	65.2	85.2	+20.0	56.8	86.7	+29.9	51.5	79.5	+28.0

Table 1: Performance before and after RL. Base and +RL denote the initial and RL-trained policies, respectively.

\Delta

denotes the absolute improvement over Base in percentage points.

Policy	Paired	No-Book	Topic-Matched	Source	Non-Source
Policy	Textbook	No-Book	Textbook	Removal	Removal
Base	51.5	55.7	49.6	49.0	53.0
+RL	79.5	71.7	75.2	74.6	79.8

Table 2: Performance under different reference conditions. Base and +RL denote the initial and Self-Learning -trained policies, respectively. No-Book disables textbook access and the tool interface, while all other conditions retain the Self-Learning evaluation protocol. Scores are overall proof accuracies across ID-Book , OOD-Book , and OOD-Subject .

RL Protocol	BrowseComp-Plus	CL-bench
RL Protocol	BrowseComp-Plus	DKR	RSA	PTE	EDS	Overall
Base	50.2	14.5	16.3	16.6	7.5	14.8
No-Book	48.6 $\blacktriangledown\,1.6$	18.4 $\blacktriangle\,3.9$	15.6 $\blacktriangledown\,0.7$	15.5 $\blacktriangledown\,1.1$	12.6 $\blacktriangle\,5.0$	16.2 $\blacktriangle\,1.4$
Top- $k$ Sections	49.9 $\blacktriangledown\,0.3$	16.1 $\blacktriangle\,1.7$	16.1 $\blacktriangledown\,0.2$	16.4 $\blacktriangledown\,0.2$	9.6 $\blacktriangle\,2.0$	15.5 $\blacktriangle\,0.7$
Long-Context	49.4 $\blacktriangledown\,0.8$	16.9 $\blacktriangle\,2.4$	13.4 $\blacktriangledown\,2.8$	15.3 $\blacktriangledown\,1.3$	11.6 $\blacktriangle\,4.0$	14.9 $\blacktriangle\,0.1$
Oracle Section	48.2 $\blacktriangledown\,2.1$	16.1 $\blacktriangle\,1.7$	13.1 $\blacktriangledown\,3.2$	14.9 $\blacktriangledown\,1.7$	11.1 $\blacktriangle\,3.5$	14.4 $\blacktriangledown\,0.4$

Table 3: Generalization to out-of-distribution tasks. Base denotes the initial policy; other rows identify the RL training protocols. Colored deltas denote gains and drops relative to Base. CL-bench categories are Domain Knowledge Reasoning (DKR), Rule System Application (RSA), Procedural Task Execution (PTE), and Empirical Discovery and Simulation (EDS).

RL Setting	Proof Accuracy				Tool Use
RL Setting	ID-Book	OOD-Book	OOD-Subject	Overall	Tool Traj.	Calls/Traj.
Base	33.9	65.2	56.8	51.5	36.9	1.5
RL	60.7	81.9	81.5	74.9	6.7	0.3
RL + Tool Credit	64.0 $\blacktriangle\,3.3$	81.1 $\blacktriangledown\,0.8$	82.5 $\blacktriangle\,1.0$	76.2 $\blacktriangle\,1.3$	7.0	0.3
RL + Exploration Bonus	61.8 $\blacktriangle\,1.1$	83.9 $\blacktriangle\,2.0$	84.4 $\blacktriangle\,2.9$	77.0 $\blacktriangle\,2.1$	79.9	5.9
Self-Learning (full)	65.5 $\blacktriangle\,4.8$	85.2 $\blacktriangle\,3.3$	86.7 $\blacktriangle\,5.2$	79.5 $\blacktriangle\,4.6$	58.4	3.0

Table 4: Ablation of RL components under the same Self-Learning protocol. Base denotes the initial policy, and RL denotes correctness-only GRPO. The full method combines tool credit and the exploration bonus. Colored deltas denote gains and drops relative to RL. Tool Traj. is the percentage of trajectories containing at least one tool call, and Calls/Traj. averages over all trajectories.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Subject	# Textbooks	# Problems	# sections/book	# tokens/book
Analysis	34	1,345	45.3	187,452
Algebra and Number Theory	33	987	42.3	177,288
Geometry and Topology	26	938	49.3	205,916
Graph Theory	12	891	63.8	199,137
Differential Equations	13	257	72.3	302,316
Numerical Computation and Optimization	13	209	81.1	269,234

Appendix

Table 5: Subject-wise statistics of our textbook–problem dataset. # sections/book and # tokens/book report per-textbook averages over the full parsed hierarchy.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 174 papers

Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models
May 8, 2026Aswin RRV, Jacob Dineen, Divij Handa +4Synthetic Data Generation RL for Language Model Reasoning
Arizona State University · Google Cloud AI Research · Google DeepMind
OpenSIR: Open-Ended Self-Improving Reasoner
Nov 1, 2025Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3Self-Play RL RL for Language Model Reasoning
University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1
Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
Jan 26, 2026Shobhita Sundaram, John Quan, Ariel Kwiatkowski +3Self-Play RL RL for Language Model Reasoning
1MIT · 2Meta FAIR · 3New York University

May 8, 2026J/K move · Enter open · S save

Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

Aswin RRV, Jacob Dineen, Divij Handa, Mihir Parmar+3

Arizona State University · Google Cloud AI Research · Google DeepMind

Synthetic Data Generation RL for Language Model Reasoning Synthetic Data Generation for Language Models LLM Training LLM Reasoning

Open paper AskPDF arXiv

The effectiveness of Reinforcement Learning (RL) in Large Language Models (LLMs) depends on the nature and diversity of the data used before and during RL. In particular, reasoning problems can often be approached in multiple ways that rely on different forms of reasoning, and exposure to only a limited range of such approaches in the training data may limit the effectiveness of RL. Motivated by this, we investigate using diverse self-generated data during mid-training as an intermediate step before RL training. Specifically, we adopt a bootstrapped data-generation framework guided by George Polya's problem-solving approaches for generating multiple variants of correct answers for each question in the training data, and then perform fine-tuning. We first provide a theoretical perspective on how mid-training on such data improves RL and explain how policy-gradient updates can incentivize combining multiple approaches. We then empirically demonstrate that RL-trained models initialized with our mid-training data achieve consistent improvements across various mathematical reasoning benchmarks and other OOD tasks like code generation and narrative reasoning. Overall, our investigative study shows that a language model learning multiple problem-solving approaches, through self-generated data helps subsequent RL.

Abstract

Language Model Self-Improvement

Explore similar work

Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

OpenSIR: Open-Ended Self-Improving Reasoner

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability