Context Bootstrapped Reinforcement Learning

Saaket Agashe; Jayanth Srinivasa; Gaowen Liu; Ramana Kompella; Xin Eric Wang

cs.LGMar 19, 2026

Context Bootstrapped Reinforcement Learning

Authors: Saaket Agashe, Jayanth Srinivasa, Gaowen Liu, Ramana Kompella, Xin Eric Wang

Organizations: University of California, Santa Barbara · Cisco Research

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) suffers from exploration inefficiency, where models struggle to generate successful rollouts, resulting in minimal learning signal. This challenge is particularly severe for tasks that require the acquisition of novel reasoning patterns or domain-specific knowledge. To address this, we propose Context Bootstrapped Reinforcement Learning (CBRL), which augments RLVR training by stochastically prepending few-shot demonstrations to training prompts. The injection probability follows a curriculum that starts high to bootstrap early exploration, then anneals to zero so the model must ultimately succeed without assistance. This forces the policy to internalize reasoning patterns from the demonstrations rather than relying on them at test time. We validate CBRL across two model families and five Reasoning Gym tasks. Our results demonstrate that CBRL consistently improves success rate, provides better exploration efficiency, and is algorithm-agnostic. We further demonstrate CBRL's practical applicability on Q, a domain-specific programming language that diverges significantly from mainstream language conventions.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the Context Bootstrapped Reinforcement Learning (CBRL) framework. (a) The injection probability annealing schedule. At each timestep $t$ , training batches consist of samples without few shots and samples with few shots , with the proportion governed by $p_{i}$ , which decreases linearly from $p_{\mathrm{start}}$ to $p_{\mathrm{end}}$ . (b) Construction of the two sample types. Samples without exemplars (top) present only the task query followed by an empty assistant turn. Samples with exemplars (bottom) prepend solved few-shot demonstrations as prior user–assistant exchanges before the target query.

Panel of Figure 2 — Figure 2: Equivalent implementations of a fixed-point search algorithm in Python (left) and Q (right). Both solutions use binary search to find an index $i$ such that $\texttt{arr}[i]=i$ . While the Python version follows conventional syntax familiar to pretrained language models, the Q implementation employs a terse, array-oriented style with right-to-left evaluation, minimal punctuation, and implicit returns.

	Qwen2.5-3B-Instruct				Llama-3.2-3B-Instruct
Environment	Baseline	Baseline w/ Few Shots	GRPO	CBRL GRPO	Baseline	Baseline w/ Few Shots	GRPO	CBRL GRPO
ARC-1D	$\phantom{0}4.67\pm 0.33$	$\phantom{0}2.00\pm 0.58$	$26.00\pm 1.15$	$\mathbf{30.67}$ $\pm\,0.67$	$\phantom{0}2.33\pm 0.33$	$\phantom{0}3.67\pm 0.88$	$17.00\pm 0.58$	$\mathbf{25.00}$ $\pm\,0.58$
Manipulate Matrix	$\phantom{0}1.33\pm 0.33$	$\phantom{0}5.00\pm 1.00$	$\phantom{0}6.00\pm 0.58$	$\phantom{0}\mathbf{8.00}$ $\pm\,0.58$	$\phantom{0}0.33\pm 0.33$	$\phantom{0}2.67\pm 0.88$	$\phantom{0}3.33\pm 0.88$	$\phantom{0}\mathbf{8.33}$ $\pm\,0.33$
Spell Backward	$\phantom{0}4.33\pm 1.33$	$\phantom{0}4.00\pm 0.58$	$51.00\pm 0.00$	$\mathbf{52.67}$ $\pm\,0.67$	$20.00\pm 3.46$	$23.67\pm 1.20$	$95.67\pm 0.33$	$\mathbf{97.00}$ $\pm\,0.58$
Word Sorting	$\phantom{0}1.33\pm 0.33$	$\phantom{0}8.67\pm 1.45$	$53.33\pm 0.67$	$\mathbf{75.67}$ $\pm\,1.20$	$\phantom{0}8.67\pm 1.67$	$13.33\pm 0.88$	$80.00\pm 0.58$	$\mathbf{84.33}$ $\pm\,0.88$
Puzzle24	$\phantom{0}4.33\pm 1.33$	$\phantom{0}6.67\pm 0.33$	$48.00\pm 1.00$	$\mathbf{60.67}$ $\pm\,0.88$	$\phantom{0}2.67\pm 0.88$	$\phantom{0}4.00\pm 2.08$	$20.67\pm 0.67$	$\mathbf{22.00}$ $\pm\,0.00$

Table 1: Success rates (%) on five Reasoning Gym environments for Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct. Baseline and Baseline w/ Few Shots evaluate the pretrained model without any RL training (zero-shot and few-shot prompting, respectively). GRPO is standard GRPO training; CBRL GRPO augments GRPO with context bootstrapping (ours). All values are mean

\pm

standard error over three seeds; bold indicates the best result per environment and model. CBRL improves over standard GRPO in all ten model–environment pairs, with gains ranging from

+1.3\%

(Spell Backward, Llama) to

+22.3\%

(Word Sorting, Qwen).

Method	Valid Q (%)	Avg. Pass (%)	Success Rate (%)
Baseline	3.1	0.3	0.0
Baseline w/ Few Shots	4.3	0.6	0.4
GRPO	89.1	27.3	5.0
CBRL GRPO	80.9	43.0	26.3

Table 2: Performance comparison on Q programming language generation for QQwen-7B-Pretrain model. All results are mean over 5 samples and values represent percentages (%). Valid Q Rate measures syntactically correct outputs; Avg. Test Pass and Success Rate measure functional correctness.

Figure 4: Training reward curves for CBRL and baseline across three settings: Q Programming with GRPO (left), Word Sorting with GRPO (center), and Word Sorting with RLOO (right). Shaded regions indicate

p_{i}>0.25

(high injection). CBRL achieves higher early reward by guiding the model toward successful rollouts, bootstrapping the learning process. The advantage persists after injection stops, demonstrating that CBRL addresses exploration inefficiency without creating long-term dependence on demonstrations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Model
Base Model	Qwen2.5-3B-Instruct
Optimization
Algorithm	GRPO / RLOO
Training Steps	500
Learning Rate	$1\times 10^{-6}$

Appendix

Table 4: Common training hyperparameters across all Reasoning Gym environments.

Hyperparameter	ARC-1D	Puzzle 24	Manipulate Matrix	Spell Backward	Word Sorting
Max Prompt Length	4,096	1,024	3,072	768	1,024
Max Response Length	2,048	512	512	256	512
Samples per Prompt ( $n$ )	4	4	4	8	4

Appendix

Table 5: Environment-specific hyperparameters for Reasoning Gym experiments.

Metric	ARC-1D	Puzzle 24	Manipulate Matrix	Spell Backward	Word Sorting
Training Steps	500	500	500	500	500
Batch Size	32	32	32	32	32
Samples per Prompt ( $n$ )	4	4	4	8	4

Appendix

Table 6: Training data consumption for Reasoning Gym experiments. Reasoning Gym environments generate problems procedurally, so we report the effective number of unique prompts seen during training.

Environment	Configuration Parameters
ARC-1D	min_size=10 , max_size=30 , num_train=3
Puzzle 24	operators={+, -, *, /} , min_value=1 , max_value=10
Manipulate Matrix	min_rows=2 , max_rows=10 , min_cols=2 , max_cols=10 ,
	min_transforms=1 , max_transforms=10
Spell Backward	min_word_len=3 , max_word_len=10
Word Sorting	min_words=3 , max_words=10 ,

Appendix

Table 7: Dataset generation configurations for each Reasoning Gym environment.

Hyperparameter	Value
ICL Injection Enabled	True
Start Probability ( $p_{\text{start}}$ )	0.5
End Probability ( $p_{\text{end}}$ )	0.0
Number of ICL Examples	2
Injection Seed	0

Appendix

Table 8: CBRL in-context learning injection hyperparameters.

Hyperparameter	Value
Temperature	0.6
Top- $p$	0.9
Evaluation Set Size	100
Evaluation Repeats	3
Developer Prompt	DeepSeekZero
Precision	bfloat16

Appendix

Table 9: Evaluation hyperparameters for Reasoning Gym experiments.

Hyperparameter	Value
Model
Base Model	qqwen-7B-Pretrain Hogan et al. (2025)
Model Size	7B parameters
Optimization
Algorithm	GRPO
Total Epochs	64

Appendix

Table 10: Training hyperparameters for Q programming experiments.

Metric	Value
Dataset
Training Set Size	542 problems
Test Set Size	136 problems
Data Source	morganstanley/sft-python-q-problems
Training Data Consumption
Batch Size	64

Appendix

Table 11: Dataset statistics and training data consumption for Q programming experiments.

Hyperparameter	Value
ICL Injection Enabled	True
Start Probability ( $p_{\text{start}}$ )	0.5
End Probability ( $p_{\text{end}}$ )	0.0
Number of ICL Examples	2
Injection Seed	42

Appendix

Table 12: CBRL in-context learning injection hyperparameters for Q programming.

Parameter	Value
Reward Type	Code Execution
Base Reward Weight	1.0
Perfect Bonus	2.0
Max Test Cases per Problem	5
Execution Timeout	10 seconds
Reward Computation

Appendix

Table 13: Reward function configuration for Q programming experiments.

Task	Expected Output Format
Reasoning Gym	<think> reasoning process </think> <answer> final answer </answer>
Q Programming	`q solve:[args]`

Appendix

Table 14: Expected model output formats for each task.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,057 papers

Boosting LLM Exploration via Weak-Model Guidance in RLVR
Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2RL for Language Model Reasoning RL for Language Models
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies
Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
May 15, 2026Chanuk Lee, Sangwoo Park, Minki Kang +1RL for Language Model Reasoning RL Exploration
KAIST
Decoupling Exploration from Optimization in RLVR
Oct 7, 2026Saif Punjwani, Micah GoldblumRL for Language Model Reasoning RL Exploration
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University

Aug 27, 2026J/K move · Enter open · S save

Boosting LLM Exploration via Weak-Model Guidance in RLVR

Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang+1

Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies

RL for Language Model Reasoning RL for Language Models Reinforcement Learning Reinforcement Learning with Verifiable Rewards LLM-Guided RL

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@

k

for large

k

. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as

k

scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.

Abstract

RL for Language Model Reasoning

Explore similar work

Boosting LLM Exploration via Weak-Model Guidance in RLVR

Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR

Decoupling Exploration from Optimization in RLVR