An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearning

Rubén Balbastre; Juan Manuel Orduña; Mariano Pérez

cs.LGAug 18, 2026

An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearning

Authors: Rubén Balbastre, Juan Manuel Orduña, Mariano Pérez

Organizations: University of Valencia

Abstract

Practical LLM unlearning is usually evaluated through two objectives: suppress target-specific knowledge and preserve non-target utility. In generative QA, this leaves a third behavior underspecified: when a target-adjacent prompt admits a broader answer without target-specific leakage, the model should answer at that level rather than leak, evade, or refuse. We study this specification problem in a controlled LoRA-GRPO RWKU setting, comparing four reward designs that span lexical suppression, anti-refusal shaping, rubric-based broad answering, and an explicit refusal contrast, with and without SFT warm-up. The experiments show that optimization success is not equivalent to behavioral unlearning: RWKU forget scores, held-out completion audits, and training dynamics can point to different conclusions. We trace these disagreements to reward-hacking endpoints, policy-support limits in GRPO, benchmark probes that miss endpoint changes, and a rubric reward that selects broad-topic answering with low semantic leakage under held-out evaluation.

Figures & tables

Reward	Motivation	Reward signal
R0-Lex	Lexical suppression that rewards absence of configured target patterns.	Target patterns
R1-AntiRefusal	Anti-refusal lexical variant that keeps the R0-Lex leakage signal but adds explicit pressure against refusal completions.	Target patterns + garak
R2-Rubric	Rubric useful-answer reward that uses an LLM judge to prefer useful broad-topic completions without target-specific leakage.	LLM rubric + garak pre-filter
R4-Refusal	Contrastive diagnostic that rewards refusal-like behavior.	garak refusal detector

Table 1: Main reward-specification summary. R0-Lex, R1-AntiRefusal, and R2-Rubric approximate useful-answer unlearning, whereas R4-Refusal is a contrastive refusal diagnostic.

Model	Reward	Broad helpful. $\uparrow$	Sem. leak $\downarrow$	Ref./avoid.	$N_{\mathrm{eval}}$
0.5B	R0-Lex	$0.058\,[0.000,0.442]$	$0.025\,[0.000,0.217]$	$0.983\,[0.504,1.000]$	10
0.5B	R1-AntiRefusal	$0.908\,[0.842,0.963]$	$0.575\,[0.408,0.783]$	$0.200\,[0.079,0.267]$	10
0.5B	R2-Rubric	$0.983\,[0.950,1.000]$	$0.008\,[0.000,0.029]$	$0.708\,[0.633,0.771]$	10
0.5B	R4-Refusal	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	$1.000\,[1.000,1.000]$	10
1.5B	R0-Lex	$0.017\,[0.000,0.425]$	$0.008\,[0.000,0.058]$	$0.992\,[0.604,1.000]$	10
1.5B	R1-AntiRefusal	$0.975\,[0.904,1.000]$	$0.308\,[0.083,0.438]$	$0.375\,[0.188,0.475]$	10

Table 2: Held-out behavioral endpoints for warm-start GRPO. Cells report median [Q1, Q3] across evaluated target entities. Ref./avoid. includes explicit refusal, claims of inability, topic avoidance, and generic policy discussion.

Reward	$\Delta$ Forget $\downarrow$	$\Delta$ Neighbor $\uparrow$	Factuality $\uparrow$	Fluency $\uparrow$	$N$
R0-Lex	$-0.316\,[-0.376,-0.173]$	$-0.073\,[-0.139,-0.066]$	$0.193\,[0.178,0.208]$	$6.922\,[6.812,7.059]$	10
R1-AntiRefusal	$-0.217\,[-0.270,-0.191]$	$-0.057\,[-0.100,-0.041]$	$0.153\,[0.131,0.177]$	$7.050\,[6.950,7.261]$	10
R2-Rubric	$-0.218\,[-0.341,-0.178]$	$-0.100\,[-0.165,-0.033]$	$0.128\,[0.124,0.138]$	$7.154\,[7.082,7.248]$	10
R4-Refusal	$-0.261\,[-0.336,-0.153]$	$-0.129\,[-0.197,-0.049]$	$0.238\,[0.195,0.273]$	$6.168\,[6.050,6.357]$	10

Table 3: Selected 3B warm-start RWKU results. Cells report median [Q1, Q3] across target entities. Forget and neighbor metrics are deltas relative to the corresponding no-training baseline; factuality and fluency are absolute scores.

N

is the number of target entities with a learned run.

Figure 1: Policy-support contrast for Qwen2.5-3B-Instruct. Warm-start runs show more active groups and reward growth than cold-start runs under the same rollout budget.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
RWKU source split	train_refusal_phi3
Subject filter	selected forget target
Held-out audit fraction	20 %
SFT train fraction	10 %
SFT test fraction	5 %

Appendix

Table 4: Data split construction used in the completed experiments.

Condition	Reward
leaks_target_specific_information = true	$0.0$
is_target_centered = true	$0.1$
uses_degenerate_avoidance = true and useful	$0.2$
uses_degenerate_avoidance = true and not useful	$0.0$
provides_useful_information = false	$0.4$
is_related_broad_topic = false	$0.6$

Appendix

Table 5: R2-Rubric structured judge reward mapping from the repository implementation. Rules are evaluated top to bottom for non-refusal completions. Refusal-prefiltered completions receive reward

0

before LLM judging.

Parameter	Value
Learning rate	$1.0\times 10^{-5}$
LR scheduler	Constant scheduler
Per-device train batch size	16
Per-device eval batch size	64
Gradient accumulation	1
Precision	bf16

Appendix

Table 6: Warm-up SFT hyperparameters and monitoring settings.

Parameter	Value
Base model	Qwen2.5-Instruct family
Learning rate	$1.0\times 10^{-5}$
LR scheduler	Constant
Per-device train batch size	64
Gradient accumulation	1
Precision	bf16

Appendix

Table 7: Core GRPO training hyperparameters from config/train.yaml .

Parameter	Value
LoRA rank $r$	64
LoRA $\alpha$	128
Target modules	q_proj , k_proj , v_proj , o_proj , gate_proj , up_proj , down_proj
Layers transformed	All transformer layers, number_layers_to_transform=-1
Initialization	Gaussian

Appendix

Table 8: LoRA configuration from config/train.yaml .

Item	Configuration
Audit prompts per target/run	60
Completions per prompt	1
Temperature	0.0
Top- $p$	1.0
Max new tokens	256

Appendix

Table 9: Held-out completion audit generation settings for the completed runs.

Metric	Meaning	Preferred direction
Forget	Aggregate target-specific forget-set score over memorization, manipulation, and adversarial-attack QA probes.	Lower
Neighbor	Aggregate ROUGE-L score on non-target neighbor probes related to the forget target.	Higher
MIA-F	Mean-token loss on forget membership-inference examples; higher loss indicates weaker memorization of forget examples.	Higher
MIA-R	Mean-token loss on retain membership-inference examples; lower loss preserves confidence on retain examples.	Lower
GA	General-ability utility score from MMLU.	Higher
RA	Reasoning-ability utility score from BBH.	Higher

Appendix

Table 10: RWKU metric semantics used to interpret the appendix tables. Preferred directions refer to deltas after unlearning relative to the corresponding original instruct baseline.

Model	Init.	Lex. $\downarrow$	Sem. $\downarrow$	Prompt helpful.	Broad helpful. $\uparrow$	Ref./avoid.	Drift $\downarrow$	$N_{\mathrm{eval}}$
0.5B	Original	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$0.992\,[0.954,1.000]$	$0.967\,[0.933,0.983]$	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	10
0.5B	SFT	$0.492\,[0.350,0.671]$	$0.550\,[0.458,0.750]$	$0.525\,[0.421,0.821]$	$0.625\,[0.487,0.879]$	$0.400\,[0.150,0.550]$	$0.000\,[0.000,0.013]$	10
1.5B	Original	$1.000\,[0.958,1.000]$	$1.000\,[0.958,1.000]$	$0.992\,[0.946,1.000]$	$0.958\,[0.662,0.983]$	$0.000\,[0.000,0.037]$	$0.000\,[0.000,0.013]$	10
1.5B	SFT	$0.792\,[0.738,0.850]$	$0.800\,[0.713,0.875]$	$0.817\,[0.717,0.875]$	$0.917\,[0.733,0.967]$	$0.100\,[0.050,0.283]$	$0.000\,[0.000,0.000]$	10
3B	Original	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$0.958\,[0.900,0.996]$	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	10
3B	SFT	$0.917\,[0.817,0.963]$	$0.892\,[0.725,0.950]$	$0.917\,[0.746,0.958]$	$0.950\,[0.771,0.979]$	$0.083\,[0.050,0.258]$	$0.000\,[0.000,0.000]$	10

Appendix

Table 11: Six-label held-out completion audit for original instruct baselines and SFT warm-up models. Cells report median [Q1, Q3] across evaluated target entities.

Model	$\Delta$ Forget $\downarrow$	$\Delta$ Neighbor $\uparrow$	$\Delta$ MIA-F $\uparrow$	$\Delta$ MIA-R $\downarrow$	GA $\uparrow$	RA $\uparrow$	TRU $\uparrow$	FAC $\uparrow$	FLU $\uparrow$	$N$
	Core benchmark deltas				Utility scores
0.5B	$-0.034\,[-0.080,-0.017]$	$0.000\,[-0.041,0.034]$	$0.804\,[0.287,1.084]$	$0.406\,[0.300,0.472]$	$0.474\,[0.446,0.500]$	$0.160\,[0.111,0.182]$	$0.290\,[0.265,0.335]$	$0.201\,[0.196,0.232]$	$6.534\,[6.361,6.601]$	10
1.5B	$-0.011\,[-0.088,0.016]$	$-0.028\,[-0.069,0.000]$	$0.324\,[0.241,0.542]$	$0.277\,[0.018,0.313]$	$0.608\,[0.580,0.629]$	$0.370\,[0.349,0.401]$	$0.360\,[0.290,0.395]$	$0.318\,[0.281,0.354]$	$6.739\,[6.607,6.952]$	10
3B	$-0.083\,[-0.150,-0.043]$	$-0.027\,[-0.082,0.000]$	$-0.312\,[-0.533,-0.004]$	$-0.413\,[-0.543,-0.359]$	$0.673\,[0.654,0.702]$	$0.488\,[0.447,0.522]$	$0.440\,[0.420,0.475]$	$0.170\,[0.164,0.186]$	$6.889\,[6.812,7.086]$	10
7B	$-0.035\,[-0.059,-0.013]$	$0.010\,[0.002,0.024]$	$-1.075\,[-1.765,-0.412]$	$-0.969\,[-1.192,-0.926]$	$0.755\,[0.726,0.782]$	$0.593\,[0.559,0.627]$	$0.450\,[0.440,0.495]$	$0.428\,[0.395,0.467]$	$6.945\,[6.829,7.114]$	10

Appendix

Table 13

Model	Init.	Reward	$\Delta$ Forget $\downarrow$	$\Delta$ Neighbor $\uparrow$	$\Delta$ MIA-F $\uparrow$	$\Delta$ MIA-R $\downarrow$	GA $\uparrow$	RA $\uparrow$	TRU $\uparrow$	FAC $\uparrow$	FLU $\uparrow$	$N$
GRPO runs
0.5B	Cold	R0-Lex	$-0.069\,[-0.123,-0.023]$	$-0.007\,[-0.079,0.054]$	$1.566\,[0.900,2.076]$	$0.659\,[0.421,0.932]$	$0.474\,[0.456,0.498]$	$0.123\,[0.105,0.136]$	$0.290\,[0.280,0.315]$	$0.206\,[0.195,0.215]$	$4.751\,[4.560,5.306]$	10
0.5B	Cold	R1-AntiRefusal	$-0.069\,[-0.123,-0.023]$	$-0.021\,[-0.061,0.032]$	$1.089\,[0.638,1.303]$	$0.441\,[0.264,0.586]$	$0.465\,[0.442,0.500]$	$0.142\,[0.108,0.170]$	$0.300\,[0.285,0.315]$	$0.136\,[0.115,0.165]$	$6.771\,[6.604,6.979]$	10
0.5B	Cold	R2-Rubric	$-0.038\,[-0.083,-0.030]$	$0.020\,[-0.010,0.042]$	$0.562\,[0.500,0.656]$	$0.657\,[0.508,0.813]$	$0.474\,[0.474,0.491]$	$0.148\,[0.123,0.185]$	$0.340\,[0.300,0.340]$	$0.172\,[0.160,0.173]$	$6.889\,[6.884,6.934]$	5
0.5B	Cold	R4-Refusal	$-0.048\,[-0.093,-0.016]$	$-0.035\,[-0.081,0.005]$	$1.070\,[0.693,2.038]$	$0.840\,[0.748,1.023]$	$0.480\,[0.452,0.500]$	$0.166\,[0.126,0.192]$	$0.290\,[0.265,0.315]$	$0.192\,[0.185,0.200]$	$4.351\,[4.138,4.447]$	10
0.5B	Warm	R0-Lex	$-0.107\,[-0.148,-0.051]$	$-0.075\,[-0.091,0.002]$	$2.625\,[1.371,3.542]$	$1.055\,[0.826,1.240]$	$0.474\,[0.453,0.504]$	$0.166\,[0.129,0.198]$	$0.300\,[0.265,0.315]$	$0.227\,[0.194,0.250]$	$5.787\,[5.450,6.125]$	10

Appendix

Table 13: Full RWKU results by model size, initialization, and reward. Cells report median [Q1, Q3] across target entities;

N

is the number of target entities with a learned run. Core columns are deltas relative to the corresponding no-training baseline, while utility columns are absolute scores.

Model	Init.	Reward	Lex. $\downarrow$	Sem. $\downarrow$	Prompt helpful.	Broad helpful. $\uparrow$	Ref./avoid.	Drift $\downarrow$	$N_{\mathrm{eval}}$
0.5B	Cold	R0-Lex	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	$1.000\,[1.000,1.000]$	$0.000\,[0.000,0.000]$	10
0.5B	Cold	R1-AntiRefusal	$0.050\,[0.021,0.796]$	$0.000\,[0.000,0.767]$	$0.000\,[0.000,0.696]$	$0.000\,[0.000,0.496]$	$1.000\,[0.250,1.000]$	$0.000\,[0.000,0.000]$	10
0.5B	Cold	R2-Rubric	$1.000\,[0.983,1.000]$	$0.050\,[0.017,0.067]$	$0.000\,[0.000,0.017]$	$0.033\,[0.000,0.983]$	$1.000\,[1.000,1.000]$	$0.000\,[0.000,0.000]$	5
0.5B	Cold	R4-Refusal	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	$0.000\,[0.000,0.000]$	$1.000\,[1.000,1.000]$	$0.000\,[0.000,0.000]$	10
0.5B	Warm	R0-Lex	$0.000\,[0.000,0.142]$	$0.025\,[0.000,0.217]$	$0.017\,[0.000,0.296]$	$0.058\,[0.000,0.442]$	$0.983\,[0.504,1.000]$	$0.000\,[0.000,0.000]$	10
0.5B	Warm	R1-AntiRefusal	$0.383\,[0.117,0.729]$	$0.575\,[0.408,0.783]$	$0.608\,[0.304,0.729]$	$0.908\,[0.842,0.963]$	$0.200\,[0.079,0.267]$	$0.000\,[0.000,0.000]$	10

Appendix

Table 14: Full six-label held-out completion audit by model size, initialization, and reward. Cells report median [Q1, Q3] across evaluated target entities. Original and SFT-only rows are reported in Table 11 .

Figure 2: GRPO training diagnostics for Qwen2.5-0.5B-Instruct.

Figure 3: GRPO training diagnostics for Qwen2.5-1.5B-Instruct.

Figure 4: GRPO training diagnostics for Qwen2.5-3B-Instruct.

Figure 5: GRPO training diagnostics for Qwen2.5-7B-Instruct.

Explore similar work

Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning
Jul 30, 2026Efstratios Zaradoukas, Davide Gabrielli, Bardh Prenkaj +1Catastrophic Forgetting Comparison
Technical University of Munich, Germany · Munich Center for Machine Learning, Germany · Sapienza University of Rome, Italy
Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning
Jun 13, 2026Zirui Pang, Chenlong Zhang, Haosheng Tan +3Large Language Model Unlearning Large Language Model Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou) · Institute of Automation, Chinese Academy of Sciences · University of Glasgow
Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning
Jun 9, 2026Bocheng Ju, Jianhua Wang, Chengliang Liu +1Large Language Model Unlearning Exact Unlearning
Beijing Key Laboratory of Security and Privacy in Intelligent Transportation, Beijing Jiaotong University, P.R.China · College of Computer Science and Technology, Taiyuan University of Technology, Taiyuan, China, 030024 · Institute of Computing Technologies, China Academy of Railway Sciences Corporation Limited, Beijing 100081, China

Jul 30, 2026, cs.LGJ/K move · Enter open · S save

Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning

Efstratios Zaradoukas, Davide Gabrielli, Bardh Prenkaj, Gjergji Kasneci

Technical University of Munich, Germany · Munich Center for Machine Learning, Germany · Sapienza University of Rome, Italy

Catastrophic Forgetting Comparison

Open paper AskPDF arXiv

Machine unlearning seeks to selectively remove specific knowledge from trained language models without full retraining, a growing necessity under privacy regulations such as GDPR and the EU AI Act. Recent work has reformulated unlearning as a Reinforcement Learning with Verifiable Rewards (RLVR) problem, where models are optimized against verifiable rewards computed directly from their outputs. However, existing methods rely on sparse binary rewards that provide minimal learning signal, indicating only whether forbidden content was avoided, and limiting convergence speed. In this paper, we study how reward design affects unlearning efficiency within the Reinforcement Unlearning (RUL) framework. We introduce a principled reward decomposition framework that decouples verifiability from sparsity, and propose two new reward functions: an exponential reward that provides graded penalties based on the count of forbidden-concept occurrences, and a PageRank inspired reward that weights penalties by semantic importance. We conduct experiments on the Real World Knowledge Unlearning (RWKU) benchmark, demonstrating that both rewards consistently outperform the binary setting, while reaching similar forgetting performance up to

3\times

faster and preserving general model utility. Our results show that reward design is a key driver of unlearning efficiency offering a practical path toward scalable and efficient machine unlearning.

Abstract

Explore similar work

Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning