Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

Rishabh Tiwari; Aditya Tomar; Udbhav Bamba; Monishwaran Maheswaran; Heng Yang; Michael W. Mahoney; Kurt Keutzer; Amir Gholami

cs.LGFeb 20, 2026

Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

Authors: Rishabh Tiwari, Aditya Tomar, Udbhav Bamba, Monishwaran Maheswaran, Heng Yang, Michael W. Mahoney, Kurt Keutzer, Amir Gholami

Organizations: UC Berkeley · Transmute AI · ICSI · LBNL

Abstract

Process Reward Models (PRMs) are rapidly becoming the backbone of LLM reasoning pipelines, yet we demonstrate that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure. To address this, we introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities. Static perturbation analysis uncovers a fluency-logic dissociation: high invariance to surface-level style changes reward changes $<$ 0.1, yet inconsistent detection of logically-corrupted reasoning, with different models failing on different attack types. Adversarial optimization demonstrates that gradient-based attacks inflate rewards on invalid trajectories, with reward landscapes exhibiting wide, exploitable peaks. RL-induced reward hacking exposes the critical failure mode: policies trained on AIME problems achieve near-perfect PRM rewards ( $>$ 0.9), while ground-truth accuracy remains low (below 4%), with 43% of reward gains attributable to stylistic shortcuts. These findings reveal that current PRMs function as fluency detectors rather than reasoning verifiers, creating systematic blind spots that undermine their use as training signals. We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment. The code and dataset are available at https://github.com/SqueezeAILab/reward-under-attack.

Figures & tables

Diagnostic Tier	Model Access	Natural Output	Optimization
Static Perturbation	None	✓	None
Adversarial Tokens	White-box	✗	Gradient
RL-Induced Reward Hacking	Black-box	✓	Policy

Table 1: Taxonomy of diagnostic tiers. “Model Access” refers to requirements for generating the attack: static perturbations are model-agnostic; adversarial tokens require gradients; and RL policies require reward queries. Tiers 1 & 3 produce natural text; Tier 2 establishes worst-case bounds.

Figure 1: Overview of static perturbation analysis. A prompt-response pair (Step 1) undergoes bias injection (Step 2), such as question shuffling where we change the question but do not modify the response (Step 3) and feed this to the PRM (Step 4). The scores are then compared against the original to quantify sensitivity (Step 5).

Figure 2: Distribution of

\Delta R

under semantics-preserving perturbations. Both PRMs exhibit tight distributions centered near zero, indicating strong invariance to surface-level stylistic changes.

Figure 3: Distribution of

\Delta R

under semantics-altering perturbations. (a) Question shuffling: Skywork penalizes mismatched questions by giving a smaller reward (peak at

\Delta R\approx-0.8

), while Qwen retains high rewards without any change. (b) Reasoning hallucination: Qwen exhibits bimodal behavior with strong penalization at

\Delta R=-1

but also substantial mass near zero which is not desirable. An ideal PRM is expected to produce very low rewards (negative

\Delta R

) for both scenarios.

Figure 4: Reward landscape for a single continuous token ( $k=1$ ) on Skywork-1.5B. A single optimized embedding vector rapidly increases mean batch reward, demonstrating that minimal adversarial capacity suffices to exploit PRM vulnerabilities.

Figure 5: Training dynamics for 100 discrete tokens on Skywork-1.5B across 8 AIME24 trajectories. Reward (blue) increases from 0.11 to 0.95 as entropy (orange) decreases, indicating successful discretization of adversarial tokens.

Panel of Figure 6 — Figure 6: Reward landscape stability analysis for Skywork-1.5B. Each plot shows PRM reward as a function of perturbations to the token sequence, averaged across 8 AIME24 trajectories. Random tokens (a, c) produce scattered, low-reward surfaces, while adversarial tokens (b, d) concentrate reward mass in wide, elevated peak. The larger basin volume around adversarial tokens (2.2 $\times$ at 100 tokens) indicates stable, exploitable regions that persist under small perturbations.

	Attack Success & Transfer				Basin Volume
$k$	AIME24	AIME25 (base)	AIME25 (+adv)	$\boldsymbol{\Delta}$	Adversarial	Random
Skywork-o1-Open-PRM-1.5B
0	0.237	0.305	-	-	-	-
1	0.289	0.305	0.335	+0.030	1.057	1.017
50	0.576	0.305	0.529	+0.224	1.372	0.853
100	0.954	0.305	0.924	+0.619	1.495	0.689

Table 2: Adversarial token optimization results. We optimize

k

discrete tokens on 8 AIME24 trajectories, and we measure transfer to 8 held-out AIME25 trajectories. The

k=0

results show the baseline (no adversarial tokens). AIME24 : best training reward achieved. AIME25 (base/+adv) : mean reward before and after appending adversarial tokens;

\Delta

is the reward change. Basin Volume : size of the high-reward region around adversarial vs. random token positions (larger = more stable exploitation).

Panel of Figure 7 — Figure 7: Reward-accuracy divergence during GRPO training. PRM reward (blue) increases while ground-truth accuracy (orange) remains flat near zero. Skywork-1.5B shows reward hacking with rewards reaching 0.8+, while Qwen-7B rewards spike to 1.0 due to mode collapse.

Figure 8: Rephrasing intervention on AIME25 for Skywork-1.5B. Distributions show rewards for base policy (orange), GRPO policy (blue), and rephrased GRPO trajectories (purple). The reward drop after rephrasing (blue

\to

purple) isolates the stylistic component of GRPO’s gains.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Distribution of $\Delta R$ for all semantics-preserving perturbations. All distributions are tightly centered near zero, indicating strong invariance to surface-level stylistic changes. Skywork-7B shows slightly broader distributions with heavier tails compared to Qwen-7B.

Panel of Figure 10 — Figure 10: Distribution of $\Delta R$ for all semantics-altering attacks. The two PRMs show divergent failure modes: Qwen-7B strongly penalizes numerical inconsistencies but misses hallucinations, while Skywork-7B shows more uniform but weaker penalization across attack types.

Figure 11: Step-by-step framework for creating the PRM-BiasBench dataset. Original prompt-response pairs are perturbed using an attack prompt via an LLM. An equivalence checker then filters out semantically altered outputs, retaining only meaning-preserving transformations. The figure illustrates this process using a rephrasing attack as an example; incorrectly altered responses are highlighted in red, while semantically equivalent responses passing the filter are shown in green.

Perturbation	Qwen2.5-Math-PRM		Skywork-o1-Open-PRM
	Mean	Std	Mean	Std
Semantics-Preserving
Rephrasing	$-0.01$	$0.03$	$-0.02$	$0.05$
Verbosity Increase	$-0.01$	$0.02$	$-0.03$	$0.06$
Verbosity Decrease	$-0.01$	$0.02$	$-0.04$	$0.07$
Reordering	$-0.03$	$0.08$	$-0.02$	$0.05$

Appendix

Table 3: Summary statistics for

\Delta R

across perturbation types. Mean and standard deviation are reported for each PRM.

Hyperparameter	Value
Data Configuration
Training Dataset	AIME 2024
Evaluation Dataset	AIME 2025
Number of Training Trajectories	8
Number of Evaluation Trajectories	8
Optimization Configuration

Appendix

Table 4: Hyperparameters for discrete adversarial token optimization.

Panel of Figure 12 — Figure 12: Reward landscape visualizations for Skywork-7B: random vs. adversarial discrete tokens, averaged across 8 AIME24 trajectories. Adversarial tokens (b, d) produce more concentrated high-reward regions compared to random tokens (a, c).

Panel of Figure 13 — Figure 13: Reward landscape visualizations for Qwen-7B: random vs. adversarial discrete tokens, averaged across 8 AIME24 trajectories. Note that for Qwen, tokens are inserted between the question and solution rather than appended.

Process Reward Models

Process reward models assess intermediate reasoning steps and actions to guide learning, search, and evaluation.

See all 140 papers

EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing
May 30, 2026Ibne Farabi Shihab, Fariya Afrin, Sanjeda Akter +1Process Reward Models Language Model Calibration
Department of Computer Science, Iowa State University · Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Civil, Construction & Environmental Engineering, Iowa State University
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
Feb 10, 2026Pei-Chi Pan, Yingbin Liang, Sen LinReward Modeling Reward Hacking
Department of Computer Science University of Houston · The Ohio State University · University of Houston
Unsupervised Process Reward Models
May 11, 2026Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2Reward Modeling Process Reward Models
Swiss Federal Institute of Technology (EPFL)

May 30, 2026J/K move · Enter open · S save

EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

Ibne Farabi Shihab, Fariya Afrin, Sanjeda Akter, Anuj Sharma

Department of Computer Science, Iowa State University · Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Civil, Construction & Environmental Engineering, Iowa State University

Process Reward Models Language Model Calibration Language Model Robustness Reward Model Evaluation

Open paper AskPDF arXiv

Process reward models (PRMs) are widely used in language-model training with dense step-level supervision. They assume PRM scores are stable proxies for step correctness under label-preserving transformations. These transformations change reasoning structure but preserve final answers. We argue this assumption is not well validated. Such transformations can change how PRM scores relate to correctness signals, leading to different failure modes across models.To address this gap, we introduce EST-PRM, a stress-testing framework for dense process rewards. It applies three transformations: (1) step inflation, (2) dependency-aware step reordering, and (3) confidence markers. A vulnerability decomposition is defined that separates reward inflation from loss of correctness sensitivity. Five PRM-style models are evaluated on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench.The results indicate clear differences in vulnerability patterns across models. Math-Shepherd shows the strongest sensitivity to position perturbations, with a Pearson correlation drop of

0.152 \pm 0.038

and a

32.8 \pm 4.9\%

score inflation rate. Qwen2.5-Math-PRM is most affected by step inflation, reaching a

47.6 \pm 4.3\%

inflation rate. Confidence-based perturbations also distort reward calibration, revealing inconsistencies in correctness estimation. Three mitigation strategies are evaluated, highlighting trade-offs between robustness coverage and false-positive rates.

Abstract

Process Reward Models

Explore similar work

EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

Unsupervised Process Reward Models