GRPO Training Dynamics for Small Language Models

Rajat Ghosh; Vaishnavi Bhargava; Henry Wong; Aryan Singhal; Debojyoti Dutta

cs.LGSep 30, 2026

GRPO Training Dynamics for Small Language Models

Authors: Rajat Ghosh, Vaishnavi Bhargava, Henry Wong, Aryan Singhal, Debojyoti Dutta

Organizations: Nutanix

Abstract

Group Relative Policy Optimization (GRPO) has emerged as a memory-efficient reinforcement fine-tuning (RFT) technique for reasoning-intensive tasks. How- ever, GRPO training dynamics on small language models (SLMs) remain poorly understood, limiting its reliable adoption and reproducibility in open and resource- constrained environments. In this work, we present a systematic study of GRPO fine-tuning for SLMs ranging from 1.5B to 7B parameters under a practical single- node 8xA100 compute budget. Our study spans multiple model families and reasoning domains, including mathematics, coding, and multiple-choice question answering (MCQ) in science. Across these settings, we analyze how group size affects policy convergence, training stability, and downstream benchmark per- formance. We further characterize tensor-level update dynamics during GRPO training and investigate whether the choice of LoRA target modules and layers can improve the performance of GRPO-tuned models. While our initial GRPO-tuned models outperform their base counterparts on approximately 80% of mathematical benchmark evaluations, they demonstrate limited capability on MCQ and code reasoning tasks. Guided by our mechanistic evaluations, we refined our LoRA and reward-shaping configurations to improve performance in latter domains. These findings provide practical guidance for GRPO training for SLMs.

Figures & tables

Figure 1 : GRPO training dynamics on GSM8K across models and group sizes (

G\in\{4,16\}

). Evaluation accuracy improves rapidly before plateauing, with DeepSeek-R1-Qwen-1.5B achieving the highest final accuracy, particularly under

G=16

. Reward and accuracy are only partially aligned: DeepSeek-7B obtains the highest training and evaluation reward, whereas DeepSeek-R1-Qwen-1.5B attains the best accuracy with lower reward. The diagnostics reveal model-dependent behavior: DeepSeek-R1-Qwen-1.5B maintains higher entropy, longer completions, lower KL, and more persistent clipping than DeepSeek-7B and Nemotron-4B.

		DeepSeek-7B			DeepSeek-R1-Qwen-1.5B			Nemotron-4B
Benchmark	Metric	Base	G=16	G=4	Base	G=16	G=4	Base	G=16	G=4
GSM-Plus (2400)	pass@1	25.64	30.83 $\pm$ 0.71	40.25 $\pm$ 0.08	33.48	36.52 $\pm$ 1.32	34.80 $\pm$ 1.50	9.11	20.32 $\pm$ 5.55	19.46 $\pm$ 2.92
GSM-Plus (2400)	pass@5	49.25	52.19 $\pm$ 0.91	60.50 $\pm$ 0.41	55.54	57.86 $\pm$ 0.67	56.44 $\pm$ 1.15	21.92	40.86 $\pm$ 8.52	42.02 $\pm$ 3.86
Math-500 (500)	pass@1	15.00	16.12 $\pm$ 0.11	15.48 $\pm$ 1.07	70.80	75.74 $\pm$ 0.59	73.18 $\pm$ 0.08	11.64	13.34 $\pm$ 0.31	12.62 $\pm$ 0.93
Math-500 (500)	pass@5	32.40	33.50 $\pm$ 0.14	33.30 $\pm$ 3.54	85.20	89.60 $\pm$ 0.85	88.70 $\pm$ 1.84	31.40	31.50 $\pm$ 0.42	31.60 $\pm$ 1.98
AIME-2026 (30)	pass@1	0.00	0.00 $\pm$ 0.00	0.34 $\pm$ 0.47	1.33	6.17 $\pm$ 0.23	4.50 $\pm$ 0.24	0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00

Table 1 : Out-of-distribution evaluation of GRPO models trained on GSM8K dataset across math benchmarks using pass@k (%). Results are reported as mean

\pm

standard deviation over random seeds. For MetaMathQA and OpenMath2 datasets, we randomly sample 2000 examples for evaluation. For NuminaMath, we skip one proof problem that is not objectively verifiable.

Figure 2 : GRPO training dynamics on ARC across models and group sizes (

G\in\{4,16\}

). Reward and accuracy are largely aligned, with Nemotron-4B performing best, followed by DeepSeek-7B and DeepSeek-R1-Qwen-1.5B. Increasing

G

yields limited gains for the stronger models but modestly improves DeepSeek-R1-Qwen-1.5B. Diagnostics reveal model-dependent regimes: Nemotron-4B exhibits low entropy with large KL movement, whereas DeepSeek-R1-Qwen-1.5B maintains high entropy, long completions, and persistent clipping.

		DeepSeek-7B			DeepSeek-R1-Qwen-1.5B			Nemotron-4B
Benchmark	Metric	Base	G=16	G=4	Base	G=16	G=4	Base	G=16	G=4
MMLU-STEM (3153)	pass@1	40.14	41.64 $\pm$ 0.25	40.97 $\pm$ 0.01	56.18	55.44 $\pm$ 0.18	55.23 $\pm$ 0.05	48.56	47.82 $\pm$ 0.05	46.92 $\pm$ 0.16
	pass@3	59.95	57.09 $\pm$ 0.66	57.82 $\pm$ 0.36	79.31	79.82 $\pm$ 0.05	79.73 $\pm$ 0.03	62.92	58.17 $\pm$ 0.09	59.64 $\pm$ 0.31
	pass@5	68.79	63.62 $\pm$ 0.64	65.16 $\pm$ 0.59	86.36	87.33 $\pm$ 0.21	87.28 $\pm$ 0.16	68.92	62.43 $\pm$ 0.11	65.23 $\pm$ 0.52
BBH (1000)	pass@1	30.82	30.83 $\pm$ 0.21	31.12 $\pm$ 0.02	40.76	41.97 $\pm$ 0.55	41.72 $\pm$ 0.16	31.40	30.67 $\pm$ 0.85	30.78 $\pm$ 0.54
	pass@3	51.17	46.78 $\pm$ 1.24	47.95 $\pm$ 0.30	69.43	69.04 $\pm$ 0.24	69.14 $\pm$ 0.12	51.45	44.80 $\pm$ 1.91	49.58 $\pm$ 0.71

Table 2 : Out-of-distribution evaluation of models trained on ARC dataset across Reasoning benchmarks using pass@k (%). Results are reported as mean

\pm

standard deviation over random seeds. For BigBenchHard (BBH), we use 4 subsets: logical_deduction_five_objects, tracking_shuffled_objects_five_objects, formal_fallacies, and temporal_sequences to ensure a diversity of questions.

		DeepSeek-7B			Nemotron-4B
Benchmark	Metric	Base	G=4	$\varepsilon_{high}=0.3$ (G=4)	Base	G=4	R. Layer (G=4)	R. Module (G=4)	$\beta=0.01$ (G=4)	All (G=4)
MMLU-STEM (3153)	pass@1	40.14	40.97	40.29	48.56	46.92	47.76	48.21	47.57	48.37
	pass@3	59.95	57.82	59.65	62.92	59.64	62.27	62.12	61.12	62.56
	pass@5	68.79	65.16	68.22	68.92	65.23	68.25	67.90	66.73	68.60
BBH (1000)	pass@1	30.82	31.12	30.98	31.40	30.78	30.66	30.80	31.04	30.68
	pass@3	51.17	47.95	50.69	51.45	49.58	51.54	50.90	50.78	49.55

Table 3 : Different hyperparameter refinement treatments for improving fine-tuning performance (pass@k score) on MCQ benchmarks. DeepSeek-7B treatment changes the clipping coefficient from

\varepsilon_{high}=0.2

0.3

. Nemotron-4B treatments include reducing LoRA fine-tuning layers (R. Layer), reducing LoRA fine-tuning modules (R. Module), increasing KL regularization from

\beta=0.003

0.01

, and combining all refinements. GRPO-baseline (

G=4

) reports the mean over random seeds. Overall, we observe hyperparameter refinements improving GRPO-tuned model performance.

Figure 3 : GRPO training dynamics on OpenCoder across models and group sizes (

G\in\{4,8\}

). Performance and reward improve rapidly before reaching diminishing returns, while larger

G

provides only model-dependent gains. KL remains bounded, entropy generally decreases, and completion lengths vary substantially across models.

		DeepSeek-7B			DeepSeek-R1-Qwen-1.5B			Nemotron-4B
Benchmark	Metric	Base	G=8	G=4	Base	G=8	G=4	Base	G=8	G=4
MBPP (500)	pass@1	34.98	34.87 $\pm$ 0.78	35.73 $\pm$ 0.21	32.38	34.46 $\pm$ 0.34	37.24 $\pm$ 0.14	32.70	33.83 $\pm$ 0.64	32.23 $\pm$ 0.30
MBPP (500)	pass@5	51.27	47.01 $\pm$ 0.72	49.40 $\pm$ 1.09	49.51	53.48 $\pm$ 0.08	54.43 $\pm$ 0.04	48.30	42.17 $\pm$ 0.57	41.29 $\pm$ 0.48
APPS (1000)	pass@1	10.23	9.33 $\pm$ 0.17	10.58 $\pm$ 0.37	20.90	26.14 $\pm$ 0.35	24.14 $\pm$ 0.16	8.79	6.59 $\pm$ 0.69	6.28 $\pm$ 0.39
APPS (1000)	pass@5	18.35	15.37 $\pm$ 0.13	16.85 $\pm$ 0.46	34.27	40.39 $\pm$ 0.33	37.48 $\pm$ 0.56	19.86	12.56 $\pm$ 1.65	13.28 $\pm$ 1.13
Codeforces (198)	pass@1	0.76	2.60 $\pm$ 0.10	1.70 $\pm$ 0.30	4.29	6.20 $\pm$ 0.10	5.10 $\pm$ 0.00	3.33	2.20 $\pm$ 0.60	2.30 $\pm$ 0.57

Table 4 : Out-of-distribution evaluation of models trained on OpenCoder dataset across coding benchmarks using pass@k (%). Results are reported as mean

\pm

standard deviation over random seeds

		DeepSeek-7B			Nemotron-4B
Benchmark	Metric	Base	G=4	R. Layer (G=4)	Base	G=4	R. Layer (G=4)	R. Layer + RR (G=4)	R. Module + RR (G=4)
MBPP (500)	pass@1	34.98	35.73	36.46	32.70	32.23	32.86	33.56	33.92
MBPP (500)	pass@5	51.27	49.40	51.44	48.30	41.29	46.47	46.85	45.99
APPS (1000)	pass@1	10.23	10.58	10.80	8.79	6.28	10.13	10.35	11.83
APPS (1000)	pass@5	18.35	16.85	17.62	19.86	13.28	20.65	20.34	20.31
Codeforces (198)	pass@1	0.76	1.70	0.76	3.33	2.30	2.83	3.18	3.03

Table 5 : Different hyperparameter refinement treatments for improving fine-tuning performance (pass@k score) on coding benchmarks. DeepSeek-7B treatment reduces LoRA fine-tuning layers (R. Layer). Nemotron-4B treatments include reducing LoRA fine-tuning layers (R. Layer), increasing weight of correctness reward on top it (R. Layer + RR), and reducing LoRA fine-tuning modules (R. Module). GRPO-baseline (G=4) reports the mean over random seeds. Overall, we observe hyperparameter refinements improving GRPO-tuned model performance.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Layer-wise parameter displacement after GRPO across model families and tasks. Each heatmap reports the per-tensor relative Frobenius change $\|W_{\mathrm{GRPO}}-W_{\mathrm{base}}\|_{F}/\|W_{\mathrm{base}}\|_{F}$ for each module and layer. Rows correspond to model families and columns correspond to reasoning datasets. GRPO induces structured rather than uniform parameter movement: updates are concentrated in attention and MLP projection tensors, while layer-normalization parameters change little. GSM8K produces the largest displacement across models, especially for DeepSeek-R1-Qwen-1.5B, whereas ARC and OpenCoder induce more localized changes.

Figure 5 : Layer-wise relative hidden-state shift induced by GRPO finetuning, base model vs. LoRA-adapted GRPO checkpoint. For each prompt

p

and transformer block

\ell

(post-block residual stream; the embedding layer is excluded), we compute

s_{p,\ell}=|(\mathbf{H}^{\text{grpo}}{p,\ell}-\mathbf{H}^{\text{base}}{p,\ell})\odot\mathbf{m}_{p}|F,/,|\mathbf{H}^{\text{base}}{p,\ell}\odot\mathbf{m}_{p}|F

, where

\mathbf{H}{p,\ell}\in\mathbb{R}^{T\times d}

is the layer-

\ell

hidden state of prompt

p

\mathbf{m}p

is the attention mask zeroing padding positions, and

|\cdot|F

is the Frobenius norm. Each panel reports the per-layer mean

\bar{s}\ell=\tfrac{1}{N}\sum_{p}s{p,\ell}

over the dataset’s full prompt set (

N{=}1172

for AI2-ARC test,

N{=}500

for MBPP test). Rows: evaluation dataset (top: AI2-ARC, bottom: MBPP). Columns: base model (left to right: DeepSeek-R1-Distill-Qwen-1.5B, Nemotron-Mini-4B-Instruct, DeepSeek-LLM-7B-Chat). Across all six combinations the shift is small and roughly flat through the bulk of the network and grows sharply in the last 2–3 layers, indicating that GRPO concentrates its representational change near the output, leaving early- and mid-stack features almost unchanged.

Panel of Figure 6 — Figure 6 : Module-wise weight update mass across models. Each subplot shows how GRPO-induced updates are distributed across transformer modules, revealing consistent patterns in specific blocks across architectures.

Component	Value	Behavior
GSM8K
Correctness Reward	2.0	Final answer matches ground truth
XML Count Reward	0.5	All required XML tags and sections are present
Numeric Answer Present Reward	0.5	Response contains a numeric answer
Format Reward	0.5	Output follows expected structure
Reasoning Present Reward	0.1	Reasoning block is non-empty

Appendix

Table 6 : Reward shaping components used in GRPO training.

Category	GSM8K	OpenCoder	ARC-Challenge
Training
Learning Rate	$1\times 10^{-5}$	$5\times 10^{-6}$	$1\times 10^{-5}$
Epochs	2	1	2
Batch Size	4	1	4
Gradient Accumulation	4	64	4
Temperature	0.8	0.8	0.8

Appendix

Table 7 : GRPO training configurations across GSM8K, OpenCoder, and ARC-Challenge.

Question. Josh decides to try flipping a house. He buys a house for $80,000 and then puts in$ 50,000 in repairs. This increased the value of the house by 150%. How much profit did he make? Ground truth: $70,000
Deepseek-LLM-7B-Chat (base) ✗	Deepseek-LLM-7B GRPO ✓
Predicted: $150,000	Predicted: $70,000
Josh bought the house for $80,000 and invested$ 50,000 in repairs, so the total cost was $80,000 +$ 50,000 = $130,000. The repairs increased the value of the house by 150%, so the new value of the house is$ 80,000 + 1.5 $\times$ $80,000 =$ 280,000 . To calculate the profit, we subtract the total cost from the new value of the house, so $280,000$ -130,000 = $150,000. Therefore, Josh made a profit of$ 150,000. The answer is: 150000	Josh bought the house for $80,000. He spent$ 50,000 on repairs. The total cost was $80,000 +$ 50,000 = $130,000. After the repairs, the value of the house increased by 150%, so the new value is$ 80,000 + ( $80,000$ \times $1.5) =$ 80,000 + $120,000 =$ 200,000. The profit Josh made is the new value of the house minus the cost, so it is $200,000$ -130,000 = $70,000. The answer is: 70000

Appendix

Table 8 : GSM8K sample where GRPO fine-tuning corrects an arithmetic error made by the base Deepseek-LLM-7B-Chat . Both GRPO seeds (G16,

s\!=\!42

and

s\!=\!3407

) recover the correct answer with the same explicit decomposition.

Question. Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market? Ground truth:$ 18
Nemotron-4B (base) ✗	Nemotron-4B GRPO ✓
Predicted: 1 (malformed)	Predicted: 18
The ducks lay 16 eggs per day. Janet eats 3 eggs per day so the number of eggs available to her is 16 $-$ 3 = 13 eggs. She bakes muffins with 4 eggs, so the number of eggs left is 13 $-$ 4 = 9 eggs. Out of these 9 eggs, she sells 9 $\times$ 2 = 18 dollars worth of eggs at the farmers’ market. Thus she makes 1} dollars every day at the farmers’ market.	Sure! Let’s break down the problem step by step. 1. Janet’s ducks lay 16 eggs per day. 2. She eats 3 eggs for breakfast every morning, so she eats 3 eggs per day. 3. She bakes muffins for her friends every day with 4 eggs. So she uses 4 eggs per day for baking. 4. In total, she uses 3 + 4 = 7 eggs per day. 5. She has 16 $-$ 7 = 9 eggs left to sell at the farmers’ market. 6. She sells 9 eggs at $2 per egg, so she makes 9$ \times $2 =$ 18 at the farmers’ market per day. Therefore, the answer is $18 .

Appendix

Table 9 : GSM8K sample where GRPO fine-tuning fixes a final-answer formatting failure in Nemotron-Mini-4B . The base reaches the correct intermediate value ($18) but emits a malformed final token 1} ; the GRPO checkpoint produces a clean final answer.

Question. Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size of Wendi’s flock is 20 chickens? Ground truth: 20
DS-R1-Qwen-1.5B (base) ✗	DS-R1-Qwen-1.5B GRPO (G4, $s\!=\!42$ ) ✓
Predicted: 25	Predicted: 20
Okay, so I need to figure out how many cups of feed Wendi needs to give her 20 chickens in the final meal of the day. Each chicken gets three cups per day, split into three meals. So total feed per day is $3\times 20=60$ cups. She already gave $15+25=40$ cups, so the remaining is $60-40=\textbf{20}$ cups. Alternatively, maybe she feeds each chicken 3 cups every day, but the 3 cups are per meal, not per chicken. If each meal is 1 cup per chicken, then for 20 chickens each meal is 20 cups. Morning: gave 15, short by 5. Afternoon: gave 25, over by 5… Wait, this is getting confusing. Maybe she can’t give 5 cups over, so she can only give 20 cups in the afternoon, but the problem says she gave 25… [oscillates between the two readings for several more paragraphs] The answer is: 25	Wendi feeds her 20 chickens three cups of mixed feed each day, divided into three meals. Total feed required per day: $3\text{ cups/chicken/day}\times 20\text{ chickens}=60\text{ cups/day}$ . Total feed given in the morning and afternoon: $15+25=40\text{ cups}$ . Subtract the feed already given from the total daily requirement: $60-40=20\text{ cups}$ . Thus, the final meal of the day requires 20 cups of feed.

Appendix

Table 10 : GSM8K sample where GRPO fine-tuning corrects a self-doubt / reasoning-loop failure in DeepSeek-R1-Distill-Qwen-1.5B . The base checkpoint reaches the correct intermediate answer (20) but oscillates between two interpretations of “three cups” and ultimately commits to the wrong final value. The GRPO checkpoint (G4,

s\!=\!42

) executes a single clean budget calculation.

Question. The period of rotation for each planet in our solar system is different. The period of rotation for Earth is: (A) 24 hours (B) 28 days (C) 13 weeks (D) 1 year Ground truth: (A)
DS-R1-Qwen-1.5B (base) ✗	DS-R1-Qwen-1.5B GRPO ✓
Predicted: (D)	Predicted: (A)
To determine the period of rotation for Earth, I need to recall the definition of a year. A year is the time it takes for Earth to complete one full orbit around the Sun Next, I should consider the options provided. Option A is 24 hours, which is the duration of a day, not a year. Option B is 28 days, which is approximately the length of a month, not a year. Option C is 13 weeks, which is about the duration of a month as well. Option D is 1 year, which aligns perfectly with the definition of a year. Therefore, the correct answer is D, 1 year. </think> To determine the period of rotation for Earth, let’s analyze the options provided: Choices: - (A) 24 hours - (B) 28 days - (C) 13 weeks - (D) 1 year Step-by-Step Explanation: 1. Understanding the Term "Period of Rotation": - The period of rotation refers to the time it takes for a celestial body to complete one full orbit around its axis. 2. Definition of a Year: - A year is the time it takes for Earth to complete one full orbit around the Sun. 3. Analyzing the Options: - Option A (24 hours): This is the duration of a day, not a year. - Option B (28 days): This is approximately the length of a month, not a year. - Option C (13 weeks): This is about the duration of a month as well. - Option D (1 year): This matches the definition of a year, as it represents Earth’s orbital period around the Sun. Conclusion: The correct answer is (D) 1 year , as it accurately describes the period of rotation for Earth. D	To determine the period of rotation for Earth, I need to recall the definition of a day. A day is the time it takes for Earth to complete one full rotation on its axis, which is 24 hours. Next, I should consider the options provided. Option A is 24 hours, which directly matches the definition of a day. Option B is 28 days, which is longer than a year and doesn’t represent Earth’s rotation period. Option C is 13 weeks, which is approximately 91 days, also not matching Earth’s rotation. Option D is 1 year, which is much longer than Earth’s rotation period. Therefore, the correct answer is A, 24 hours. </think> To determine the period of rotation for Earth, let’s analyze the options step by step. 1. Understanding the Question: - The question asks for the period of rotation of Earth. - The options are: - (A) 24 hours - (B) 28 days - (C) 13 weeks - (D) 1 year 2. Recalling the Definition of a Day: - A day is defined as the time it takes for Earth to complete one full rotation on its axis. - This duration is exactly 24 hours . 3. Evaluating the Options: - Option A (24 hours): This matches the definition of a day, making it the correct answer. - Option B (28 days): This is longer than a year and does not represent Earth’s rotation period. - Option C (13 weeks): This is approximately 91 days, which is not the period of Earth’s rotation. - Option D (1 year): This is much longer than Earth’s rotation period. 4. Conclusion: - The only option that accurately represents Earth’s period of rotation is 24 hours . A

Appendix

Table 11 : ARC sample where GRPO fine-tuning corrects a conceptual confusion between rotation and revolution. The base model conflates Earth’s orbital period with its rotational period, while the GRPO checkpoint correctly identifies axial rotation.

Question. Petrified palm trees are found in sedimentary rock near glaciers. The presence of the petrified palm trees most likely provides evidence for which statement? (A) There was once more water in the area. (B) The area was once grassland. (C) The climate in the area was once tropical. (D) There are active faults in the area. Ground truth: (C)
DeepSeek-LLM-7B-Chat (base) ✗	GRPO model ✓
Predicted: (A)	Predicted: (C)
The correct answer is (A) There was once more water in the area. The presence of petrified palm trees near glaciers suggests that the area was once near a tropical forest. This would indicate that there was once more water in the area, which is conducive to the growth of palm trees. ... (C) The climate in the area was likely not tropical at the time the palm trees were alive, as they are now fossilized.	The presence of petrified palm trees in the vicinity of a glacier indicates that the area was once warmer and wetter, which is consistent with a tropical climate. This supports the statement that the area was once tropical. Therefore, the correct answer is (C) The climate in the area was once tropical.

Appendix

Table 12 : ARC sample where GRPO fine-tuning corrects a reasoning ambiguity. The base model misattributes evidence to water presence, while the GRPO checkpoint correctly identifies climatic implication.

Question. The stem is an important part of many plants. Which of the following is most similar to the role performed by the stem of a plant? (A) an anchor holding a boat in place (B) a snack company producing energy bars (C) a sign attracting people (D) an elevator transporting supplies from one floor to another. Ground truth: (D)
Nemotron-Mini-4B (base) ✗	GRPO model ✓
Predicted: (A)	Predicted: (D)
The answer is A) an anchor holding a boat in place	The answer is (D) an elevator transporting supplies from one floor to another The stem of a plant performs a similar function as an elevator in a building. It transports materials (water and nutrients) from the roots to the leaves and other parts of the plant, just like how an elevator carries people and supplies from one floor to another. Here’s a brief comparison: - Stem (plant): Transports water and nutrients from roots to leaves and other parts of the plant. - Elevator (building): Transports people and supplies from one floor to another. Option (A) is incorrect because an anchor holds a boat in place, not performs a transport function. Option (B) is incorrect because a snack company does not produce materials. Option (C) is incorrect because a sign does not transport materials or people.

Appendix

Table 13 : ARC sample where GRPO fine-tuning improves analogical reasoning by correctly identifying functional similarity rather than superficial association. The base model produces a shallow incorrect choice with no reasoning, while the GRPO checkpoint identifies the correct transport-function mapping with structured explanation.

Question. Write a python function to find the maximum sum of elements of list in a list of lists.
Nemotron Base ✗	Nemotron GRPO ✓
Pass Rate: 0/10	Pass Rate: 7/10
$<$ reasoning $>$ To find the maximum sum of elements in a list of lists, we can iterate through each sublist and find the maximum sum for that sublist. We can do this by iterating through each element in the sublist and adding it to a running total, and then updating the maximum sum if necessary. $<$ /reasoning $>$ $<$ answer $>$ def maximum_Sum(matrix): max_sum = 0 for sublist in matrix: current_sum = 0 for num in sublist: current_sum += num if current_sum $>$ max_sum: max_sum = current_sum return max_sum $<$ /answer $>$	$<$ reasoning $>$ This problem can be solved using a dictionary to store the elements and their indices. If an element is already in the dictionary, it means it has been seen before, so we return its index. If not, we add it to the dictionary and return -1. $<$ /reasoning $>$ $<$ answer $>$ def find_first_duplicate(arr): return next((i for i, x in enumerate(arr) if arr.count(x) $>$ 1), -1) $<$ /answer $>$

Appendix

Table 14 : MBPP Task 23 sample comparing Nemotron-Mini-4B base and GRPO models.

Question. Write a function to perform index wise multiplication of tuple elements in the given two tuples.
Deepseek 7B Base ✗	Deepseek 7B GRPO ✓
Pass Rate: 0/10	Pass Rate: 10/10
$<$ reasoning $>$ The function takes two tuples as input and returns a new tuple where the i-th element is the product of the i-th elements of the two input tuples. $<$ /reasoning $>$ $<$ answer $>$ def index_multiplication(t1, t2): result = [] for i in range(len(t1)): result.append(t1[i] * t2[i]) return tuple(result) $<$ /answer $>$	$<$ reasoning $>$ The index_multiplication function takes two tuples as input and returns a new tuple with the index-wise multiplication of the elements of the input tuples. $<$ /reasoning $>$ $<$ answer $>$ def index_multiplication(t1, t2): return tuple(tuple(i * j for i, j in zip(t1[i], t2[i])) for i in range(len(t1))) $<$ /answer $>$

Appendix

Table 15 : MBPP Task 445 sample comparing Deepseek 7B base and GRPO models.

Question. Write a python function to check whether the given number can be represented as difference of two squares or not.
DeepSeek-R1-Qwen-1.5B Base ✗	DeepSeek-R1-Qwen-1.5B GRPO ✓
Pass Rate: 0/10	Pass Rate: 8/10
Okay, I need to write a Python function called dif_Square that checks whether a given number can be represented as difference of two squares or not. Let’s think about how to approach this. First, I should understand what it means for a number to be expressed as the difference of two squares. The difference of two squares can be written as a 2 - b 2 ... [... thinking overflow truncated for brevity ...] Ah, because in the code, D must be less than S. In this case, D=1, S=1. So, D is not less than S.	Alright, I need to write a Python function called dif_Square that checks if a given number can be represented as the difference of two squares. The function should return True if it can, and False otherwise. Let’s think about how to approach this. [... thinking overflow truncated for brevity ...] ### Solution Code: ‘‘‘python def dif_Square(n): if n $<$ = 0: return False for x in range(1, int(n**0.5) + 1): if n % x == 0: y = n // x if (x % 2) == (y % 2): return True return False ‘‘‘

Appendix

Table 16 : MBPP Task 72 sample comparing DeepSeek-R1-Qwen-1.5B base and GRPO models.

Question. You are given two huge binary integer numbers $a$ and $b$ of lengths $n$ and $m$ respectively. You will repeat the following process: if $b>0$ , then add to the answer the value $a~\&~b$ and divide $b$ by $2$ rounding down, and repeat the process again… Your task is to calculate the answer modulo $998244353$ .
Nemotron Base ✗	Nemotron GRPO ✓
Pass Rate: 0/10	Pass Rate: 7/10
$<$ reasoning $>$ We can solve this problem using a while loop. We iterate through the binary numbers $a$ and $b$ from right to left, performing a bitwise AND operation on them and adding the result to the answer... $<$ /reasoning $>$ $<$ answer $>$ ‘‘‘python def add_binary(a, b): n = len(a) m = len(b) ans = 0 while m $>$ 0: ans = (ans $<$ $<$ 1) + (a[-1] - ’0’) & (b[-1] - ’0’) a = a[:-1] b = b[:-1] m -= 1 return ans % 998244353 n, m = map(int, input().split()) a = input() b = input() print(add_binary(a, b)) ‘‘‘ $<$ /answer $>$	$<$ reasoning $>$ The problem can be solved by repeatedly computing the bitwise AND of $a$ and $b$ , adding the result to the answer, and then dividing $b$ by $2$ and rounding down. We need to ensure that the result is added modulo $998244353$ . $<$ /reasoning $>$ $<$ answer $>$ ‘‘‘python def add_and_divide(a, b): ans = 0 while b $>$ 0: ans += (a & b) % 998244353 b //= 2 return ans % 998244353 n, m = map(int, input().split()) a = int(input(), 2) b = int(input(), 2) print(add_and_divide(a, b)) ‘‘‘ $<$ /answer $>$

Appendix

Table 17 : APPS-Intro Task 4524 sample comparing Nemotron base and GRPO models.

Question. Takahashi is organizing a party. At the party, each guest will receive one or more snack pieces. Takahashi predicts that the number of guests at this party will be A or B. Find the minimum number of pieces that can be evenly distributed to the guests in both of the cases predicted.
Deepseek 7B Base ✗	Deepseek 7B GRPO ✓
Pass Rate: 0/10	Pass Rate: 9/10
$<$ reasoning $>$ To minimize the number of snack pieces, we need to distribute them in a way that maximizes the number of guests they can be evenly distributed to. In the worst-case scenario, we need to distribute the pieces to both A guests and B guests. Therefore, we need to find the least common multiple (LCM) of A and B... $<$ /reasoning $>$ $<$ answer $>$ def main(): A, B = map(int, input().split()) lcm = abs(A*B) // math.gcd(A, B) print(lcm) main() $<$ /answer $>$	$<$ reasoning $>$ We need to find the smallest cost of a route that takes not longer than time T. We can sort the routes by time and then select the one with the smallest cost. $<$ /reasoning $>$ $<$ answer $>$ def main(): n, t = map(int, input().split()) routes = sorted([list(map(int, input().split())) for _ in range(n)], key=lambda x: x[1]) min_cost = t for c, t in routes: if t $<$ = t: min_cost = min(min_cost, c) if min_cost == t: print(’TLE’) else: print(min_cost) main() $<$ /answer $>$

Appendix

Table 18 : APPS-Intro Task 4176 sample comparing Deepseek 7B base and GRPO models.

Question. A triple of numbers is said to be poor when two of those numbers are equal but the other number is different from those two numbers. You will be given three integers A, B, and C. If this triple is poor, print Yes; otherwise, print No.
DeepSeek-R1-Qwen-1.5B Base ✗	DeepSeek-R1-Qwen-1.5B GRPO ✓
Pass Rate: 0/10	Pass Rate: 9/10
Okay, I need to solve this problem where I have to determine if a triple of numbers is "poor." Let me read the problem statement again to make sure I understand it correctly. So, a triple is considered poor if two of the numbers are equal, but the third is different. For example... [... thinking overflow truncated for brevity ...] Another test case: 5,5,5,5,5,5,5,5,5,5 $\rightarrow$ len is 1 $\rightarrow$ no. Another test case: 5,5,5,5,5,5,5,5,5,5,5 $\rightarrow$ len is 1 $\rightarrow$ no.	Okay, I need to solve this problem where I have to determine if a given triple of numbers is ’poor’. Let’s see what the problem says. So, a triple is considered poor if exactly two of the numbers are equal, and the third is different. That means, for example, if the numbers are 5,7,5, then two are 5 and one is 7. So it’s poor. But if all three are the same, like 5,5,5, then it’s not poor. [... thinking overflow truncated for brevity ...] ### Solution Code ‘‘‘python from collections import Counter # Read the input a, b, c = map(int, input().split()) # Create a frequency counter freq = Counter([a, b, c]) # Determine the maximum frequency and the sum of the remaining frequencies max_freq = max(freq.values()) sum_remaining = sum(freq.values()) - max_freq # Check if the triple is poor if max_freq == 2 and sum_remaining == 1: print("Yes") else: print("No") ‘‘‘

Appendix

Table 19 : APPS-Intro Task 4288 sample comparing DeepSeek-R1-Qwen-1.5B base and GRPO models.

Explore similar work

May 29, 2026cs.LG

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

We identify a new dimension for enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. While GRPO relies on diverse rollouts, prevailing strategies primarily increase diversity by injecting more token-level randomness, which may introduce step-wise noise and lead to incoherent trajectories. We uncover that smaller models within the same model family inherently exhibit higher policy-level diversity, indicated by their superior pass@k relative to larger counterparts as sample counts increase. Unlike token-level noise, this diversity is temporally correlated, preserves logical consistency, and provides structured exploration signals for gradient estimation. We thus propose S2L-PO (Small-to-Large Policy Optimization), a framework that leverages fixed small models as natural explorers to train larger models. To balance exploration and exploitation, we design a progressive annealing strategy that transitions from offline small-model rollouts to the large learner's own sampling. This shift elegantly avoids mid-training performance drops caused by the small model's capacity limits, achieving faster convergence and unlocking a higher performance ceiling. S2L-PO improves accuracy on diverse mathematical reasoning benchmarks (e.g., +8.8% on AIME 24 using a 1.7B explorer to guide the 8B model) while reducing rollout compute.

Yiming Ren, Yiran Xu, Zicheng Lin +8

Tsinghua University · Shanghai AI Laboratory · The Chinese University of Hong Kong +1

Frictive Policy Optimization Rollout Small Reasoning Model Diversity Exploration

Ask PDF

May 17, 2026cs.LG

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.

Minghao Tian, Yunfei Xie, Chen Wei

Rice University

Large Language Model Reinforcement Learning Flow-Grpo Offline Reinforcement Learning Off-Policy Learning

Ask PDF

Jul 27, 2026cs.AI

Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model (

\text{PPL}<20

) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released

^§

Md Rezwanul Haque, Md. Milon Islam, Fakhri Karray

Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo, N2L 3G1, Ontario, Canada · Department of Computer Science and Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

Large Language Model Reinforcement Learning Small Large Language Models Offline Reinforcement Learning Proximal Policy Optimization Large Language Model Agents

Ask PDF