Context Bootstrapped Reinforcement Learning

Saaket Agashe; Jayanth Srinivasa; Gaowen Liu; Ramana Kompella; Xin Eric Wang

cs.LGMar 19, 2026

Context Bootstrapped Reinforcement Learning

Authors: Saaket Agashe, Jayanth Srinivasa, Gaowen Liu, Ramana Kompella, Xin Eric Wang

Organizations: University of California, Santa Barbara · Cisco Research

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) suffers from exploration inefficiency, where models struggle to generate successful rollouts, resulting in minimal learning signal. This challenge is particularly severe for tasks that require the acquisition of novel reasoning patterns or domain-specific knowledge. To address this, we propose Context Bootstrapped Reinforcement Learning (CBRL), which augments RLVR training by stochastically prepending few-shot demonstrations to training prompts. The injection probability follows a curriculum that starts high to bootstrap early exploration, then anneals to zero so the model must ultimately succeed without assistance. This forces the policy to internalize reasoning patterns from the demonstrations rather than relying on them at test time. We validate CBRL across two model families and five Reasoning Gym tasks. Our results demonstrate that CBRL consistently improves success rate, provides better exploration efficiency, and is algorithm-agnostic. We further demonstrate CBRL's practical applicability on Q, a domain-specific programming language that diverges significantly from mainstream language conventions.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the Context Bootstrapped Reinforcement Learning (CBRL) framework. (a) The injection probability annealing schedule. At each timestep $t$ , training batches consist of samples without few shots and samples with few shots , with the proportion governed by $p_{i}$ , which decreases linearly from $p_{\mathrm{start}}$ to $p_{\mathrm{end}}$ . (b) Construction of the two sample types. Samples without exemplars (top) present only the task query followed by an empty assistant turn. Samples with exemplars (bottom) prepend solved few-shot demonstrations as prior user–assistant exchanges before the target query.

Panel of Figure 2 — Figure 2: Equivalent implementations of a fixed-point search algorithm in Python (left) and Q (right). Both solutions use binary search to find an index $i$ such that $\texttt{arr}[i]=i$ . While the Python version follows conventional syntax familiar to pretrained language models, the Q implementation employs a terse, array-oriented style with right-to-left evaluation, minimal punctuation, and implicit returns.

	Qwen2.5-3B-Instruct				Llama-3.2-3B-Instruct
Environment	Baseline	Baseline w/ Few Shots	GRPO	CBRL GRPO	Baseline	Baseline w/ Few Shots	GRPO	CBRL GRPO
ARC-1D	$\phantom{0}4.67\pm 0.33$	$\phantom{0}2.00\pm 0.58$	$26.00\pm 1.15$	$\mathbf{30.67}$ $\pm\,0.67$	$\phantom{0}2.33\pm 0.33$	$\phantom{0}3.67\pm 0.88$	$17.00\pm 0.58$	$\mathbf{25.00}$ $\pm\,0.58$
Manipulate Matrix	$\phantom{0}1.33\pm 0.33$	$\phantom{0}5.00\pm 1.00$	$\phantom{0}6.00\pm 0.58$	$\phantom{0}\mathbf{8.00}$ $\pm\,0.58$	$\phantom{0}0.33\pm 0.33$	$\phantom{0}2.67\pm 0.88$	$\phantom{0}3.33\pm 0.88$	$\phantom{0}\mathbf{8.33}$ $\pm\,0.33$
Spell Backward	$\phantom{0}4.33\pm 1.33$	$\phantom{0}4.00\pm 0.58$	$51.00\pm 0.00$	$\mathbf{52.67}$ $\pm\,0.67$	$20.00\pm 3.46$	$23.67\pm 1.20$	$95.67\pm 0.33$	$\mathbf{97.00}$ $\pm\,0.58$
Word Sorting	$\phantom{0}1.33\pm 0.33$	$\phantom{0}8.67\pm 1.45$	$53.33\pm 0.67$	$\mathbf{75.67}$ $\pm\,1.20$	$\phantom{0}8.67\pm 1.67$	$13.33\pm 0.88$	$80.00\pm 0.58$	$\mathbf{84.33}$ $\pm\,0.88$
Puzzle24	$\phantom{0}4.33\pm 1.33$	$\phantom{0}6.67\pm 0.33$	$48.00\pm 1.00$	$\mathbf{60.67}$ $\pm\,0.88$	$\phantom{0}2.67\pm 0.88$	$\phantom{0}4.00\pm 2.08$	$20.67\pm 0.67$	$\mathbf{22.00}$ $\pm\,0.00$

Table 1: Success rates (%) on five Reasoning Gym environments for Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct. Baseline and Baseline w/ Few Shots evaluate the pretrained model without any RL training (zero-shot and few-shot prompting, respectively). GRPO is standard GRPO training; CBRL GRPO augments GRPO with context bootstrapping (ours). All values are mean

\pm

standard error over three seeds; bold indicates the best result per environment and model. CBRL improves over standard GRPO in all ten model–environment pairs, with gains ranging from

+1.3\%

(Spell Backward, Llama) to

+22.3\%

(Word Sorting, Qwen).

Method	Valid Q (%)	Avg. Pass (%)	Success Rate (%)
Baseline	3.1	0.3	0.0
Baseline w/ Few Shots	4.3	0.6	0.4
GRPO	89.1	27.3	5.0
CBRL GRPO	80.9	43.0	26.3

Table 2: Performance comparison on Q programming language generation for QQwen-7B-Pretrain model. All results are mean over 5 samples and values represent percentages (%). Valid Q Rate measures syntactically correct outputs; Avg. Test Pass and Success Rate measure functional correctness.

Figure 4: Training reward curves for CBRL and baseline across three settings: Q Programming with GRPO (left), Word Sorting with GRPO (center), and Word Sorting with RLOO (right). Shaded regions indicate

p_{i}>0.25

(high injection). CBRL achieves higher early reward by guiding the model toward successful rollouts, bootstrapping the learning process. The advantage persists after injection stops, demonstrating that CBRL addresses exploration inefficiency without creating long-term dependence on demonstrations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Model
Base Model	Qwen2.5-3B-Instruct
Optimization
Algorithm	GRPO / RLOO
Training Steps	500
Learning Rate	$1\times 10^{-6}$

Appendix

Table 4: Common training hyperparameters across all Reasoning Gym environments.

Hyperparameter	ARC-1D	Puzzle 24	Manipulate Matrix	Spell Backward	Word Sorting
Max Prompt Length	4,096	1,024	3,072	768	1,024
Max Response Length	2,048	512	512	256	512
Samples per Prompt ( $n$ )	4	4	4	8	4

Appendix

Table 5: Environment-specific hyperparameters for Reasoning Gym experiments.

Metric	ARC-1D	Puzzle 24	Manipulate Matrix	Spell Backward	Word Sorting
Training Steps	500	500	500	500	500
Batch Size	32	32	32	32	32
Samples per Prompt ( $n$ )	4	4	4	8	4

Appendix

Table 6: Training data consumption for Reasoning Gym experiments. Reasoning Gym environments generate problems procedurally, so we report the effective number of unique prompts seen during training.

Environment	Configuration Parameters
ARC-1D	min_size=10 , max_size=30 , num_train=3
Puzzle 24	operators={+, -, *, /} , min_value=1 , max_value=10
Manipulate Matrix	min_rows=2 , max_rows=10 , min_cols=2 , max_cols=10 ,
	min_transforms=1 , max_transforms=10
Spell Backward	min_word_len=3 , max_word_len=10
Word Sorting	min_words=3 , max_words=10 ,

Appendix

Table 7: Dataset generation configurations for each Reasoning Gym environment.

Hyperparameter	Value
ICL Injection Enabled	True
Start Probability ( $p_{\text{start}}$ )	0.5
End Probability ( $p_{\text{end}}$ )	0.0
Number of ICL Examples	2
Injection Seed	0

Appendix

Table 8: CBRL in-context learning injection hyperparameters.

Hyperparameter	Value
Temperature	0.6
Top- $p$	0.9
Evaluation Set Size	100
Evaluation Repeats	3
Developer Prompt	DeepSeekZero
Precision	bfloat16

Appendix

Table 9: Evaluation hyperparameters for Reasoning Gym experiments.

Hyperparameter	Value
Model
Base Model	qqwen-7B-Pretrain Hogan et al. (2025)
Model Size	7B parameters
Optimization
Algorithm	GRPO
Total Epochs	64

Appendix

Table 10: Training hyperparameters for Q programming experiments.

Metric	Value
Dataset
Training Set Size	542 problems
Test Set Size	136 problems
Data Source	morganstanley/sft-python-q-problems
Training Data Consumption
Batch Size	64

Appendix

Table 11: Dataset statistics and training data consumption for Q programming experiments.

Hyperparameter	Value
ICL Injection Enabled	True
Start Probability ( $p_{\text{start}}$ )	0.5
End Probability ( $p_{\text{end}}$ )	0.0
Number of ICL Examples	2
Injection Seed	42

Appendix

Table 12: CBRL in-context learning injection hyperparameters for Q programming.

Parameter	Value
Reward Type	Code Execution
Base Reward Weight	1.0
Perfect Bonus	2.0
Max Test Cases per Problem	5
Execution Timeout	10 seconds
Reward Computation

Appendix

Table 13: Reward function configuration for Q programming experiments.

Task	Expected Output Format
Reasoning Gym	<think> reasoning process </think> <answer> final answer </answer>
Q Programming	`q solve:[args]`

Appendix

Table 14: Expected model output formats for each task.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,057 papers

Aug 27, 2026

Boosting LLM Exploration via Weak-Model Guidance in RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@

k

for large

k

. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as

k

scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.

Xingyu Shen, Huishuai Zhang, Peng Li +2

Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies

RL for Language Model Reasoning RL for Language Models Reinforcement Learning Reinforcement Learning with Verifiable Rewards LLM-Guided RL

Ask PDF

May 15, 2026

Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR

Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models. However, its effectiveness is fundamentally limited by exploration: the policy can only improve on trajectories it has already sampled. While increasing the number of rollouts alleviates this issue, such brute-force scaling is computationally expensive, and existing approaches that modify the optimization objective provide limited control over what is explored. In this work, we propose NudgeRL, a framework for structured and diversity-driven exploration in RLVR. Our approach introduces Strategy Nudging, which conditions each rollout on lightweight, strategy-level contexts to induce diverse reasoning trajectories without relying on expensive oracle supervision. To effectively learn from such structured exploration, we further propose a unified objective, which decomposes the reward signal into inter- and intra-context components and incorporates a distillation objective to transfer discovered behaviors back to the base policy. Empirically, NudgeRL outperforms standard GRPO with up to 8 times larger rollout budgets, while outperforming oracle-guided RL baseline on average across five challenging math benchmarks. These results demonstrate that structured, context-driven exploration can serve as an efficient and scalable alternative to both brute-force rollout scaling and feasibility-oriented methods based on privileged information. Our code is available at https://github.com/tally0818/NudgeRL.

Chanuk Lee, Sangwoo Park, Minki Kang +1

KAIST

RL for Language Model Reasoning RL Exploration Reinforcement Learning with Verifiable Rewards

Ask PDF

Oct 7, 2026

Decoupling Exploration from Optimization in RLVR

Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@

k

scaling, indicating that ExpDis produces models that generate more diverse correct solutions.

Saif Punjwani, Micah Goldblum

Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University

RL for Language Model Reasoning RL Exploration Reinforcement Learning with Verifiable Rewards

Ask PDF

Abstract

RL for Language Model Reasoning

Explore similar work

Boosting LLM Exploration via Weak-Model Guidance in RLVR

Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR

Decoupling Exploration from Optimization in RLVR