Beyond Semantic Manipulation: Token-Space Attacks on Reward Models

Yuheng Zhang; Mingyue Huo; Minghao Zhu; Mengxue Zhang; Nan Jiang

cs.LGApr 3, 2026

Beyond Semantic Manipulation: Token-Space Attacks on Reward Models

Authors: Yuheng Zhang, Mingyue Huo, Minghao Zhu, Mengxue Zhang, Nan Jiang

Organizations: UIUC · Independent Researcher · University of Massachusetts Amherst

Abstract

Reward models (RMs) are widely used as optimization targets in reinforcement learning from human feedback (RLHF), yet they remain vulnerable to reward hacking. Existing attacks mainly operate within the semantic space, constructing human-readable adversarial outputs that exploit RM biases. In this work, we introduce a fundamentally different paradigm: Token Mapping Perturbation Attack (TOMPA), a framework that performs adversarial optimization directly in token space. By bypassing the standard decode-re-tokenize interface between the policy and the reward model, TOMPA enables the attack policy to optimize over raw token sequences rather than coherent natural language. Using only black-box scalar feedback, TOMPA automatically discovers non-linguistic token patterns that elicit extremely high rewards across multiple state-of-the-art RMs. Specifically, when targeting Skywork-Reward-V2-Llama-3.1-8B, TOMPA nearly doubles the reward of GPT-5 reference answers and outperforms them on 98.0% of prompts. Despite these high scores, the generated outputs degenerate into nonsensical text, revealing that RMs can be systematically exploited beyond the semantic regime and exposing a critical vulnerability in current RLHF pipelines.

Figures & tables

Figure 1: The TOMPA attack pipeline. The attack bypasses the standard decode–re-tokenize interface by applying a perturbation mapping, directly feeding transformed token sequences into the reward model. Trained via reinforcement learning using only scalar reward feedback, the policy automatically discovers adversarial token patterns that receive anomalously high rewards. Despite outperforming GPT-5 reference answers, the resulting outputs collapse into non-linguistic sequences — revealing a critical vulnerability in reward models that lies beyond the semantic regime.

Category	Work	Method	Content	Access
Length bias	Singhal et al. (2023) ; Shen et al. (2023) ; Park et al. (2024)	Manual	Readable	Black-box
Sycophancy bias	Sharma et al. (2023) ; Wen et al. (2024)	Manual	Readable	Black-box
Semantic manipulation	Wang et al. (2024) ; Zhao et al. (2025) ; Wang et al. (2025a)	Manual	Readable	Black-box
Template injection	Zheng et al. (2024)	Manual	Non-linguistic	Black-box
Semantic manipulation	Raina et al. (2024)	Gradient	Readable	Gray-box
Semantic OOD exploit	Bukharin et al. (2025)	RL	Readable	Black-box

Table 1: Prior work on reward model and LLM judge vulnerabilities, categorized by attack method, adversarial content type, and access requirement. Our work is the only approach that leverages RL to discover non-linguistic adversarial content while requiring only black-box access to the target reward model.

Reward Model	Method	Min	Mean	Max	Beat Gold
Qwen3-8B RM (RewardBench2 #6)	Gold Answer (GPT-5)	$+8.12$			—
	Random OOD	$-9.05$	$-7.94$	$-6.87$	$1.0\%$
	TOMPA (Ours)	$+15.86$	$+16.86$	$+17.78$	$\mathbf{98.0\%}$
Llama-3.1-8B RM (RewardBench2 #1)	Gold Answer (GPT-5)	$+17.48$			—
	Random OOD	$-4.16$	$-3.42$	$-2.33$	$0.0\%$
	TOMPA (Ours)	$+32.16$	$+33.64$	$+35.01$	$\mathbf{98.0\%}$

Table 2: Reward scores on the NB-curated evaluation set. For each prompt, we compute the min, mean, and max reward across its 8 rollouts; values are averaged across all 100 prompts based on these per-group statistics. Beat Gold denotes the percentage of rollouts whose reward exceeds the gold answer score.

Figure 2: Qualitative examples of generated sequences under our attack, presented exactly as decoded by the respective reward model’s tokenizer. Despite receiving anomalously high rewards of +21.38 for the Qwen3-8B RM and +36.75 for the Llama-3.1-8B RM, which far exceed their respective GPT-5 gold scores of +9.44 and +15.06, the outputs degenerate into cross-lingual gibberish, code snippets, and reserved tokenizer artifacts. This demonstrates that the reward models assign extremely high scores to token sequences completely devoid of semantic meaning.

Figure 3: Impact of response length on reward scores. Responses are truncated at various intervals and evaluated using the target reward models. Despite consisting of highly repetitive patterns, the reward does not scale linearly with length. Instead, both models assign low or negative scores to shorter truncated versions, before exhibiting anomalous reward surges as the sequences approach the maximum length of 2048 tokens.

Figure 4: Training curves of attack policy optimization under token mapping perturbation. Mean and maximum rewards are plotted over training steps. The attack policy initially receives strongly negative rewards, but progressively identifies high-reward patterns and eventually surpasses the GPT-5 reference answers (the horizontal dashed line).

Adversarial Attacks

Adversarial attacks exploit crafted inputs, poisoned data, or system interactions to manipulate AI predictions, behavior, privacy, or reliability.

See all 447 papers

Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models
Feb 20, 2026Rishabh Tiwari, Aditya Tomar, Udbhav Bamba +5Process Reward Models Reward Hacking
UC Berkeley · Transmute AI · ICSI +1
RMB: Reward Model Boosting Mitigates Reward Hacking
Sep 27, 2026Jiabin Fan, Dezhi Ye, Yongchang Hao +1Reward Hacking RL from Human Feedback
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Tencent · Canada CIFAR AI Chair
HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
Jun 2, 2026Shuang Liu, Yuxuan Bo, Qiuyang Zhao +4Reward Modeling Reward Hacking
Carnegie Mellon University · University of Virginia · Harvard University +4

Feb 20, 2026J/K move · Enter open · S save

Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

Rishabh Tiwari, Aditya Tomar, Udbhav Bamba, Monishwaran Maheswaran+4

UC Berkeley · Transmute AI · ICSI+1

Process Reward Models Reward Hacking Adversarial Attacks Reasoning Verification RL for Language Model Reasoning

Open paper AskPDF arXiv

Process Reward Models (PRMs) are rapidly becoming the backbone of LLM reasoning pipelines, yet we demonstrate that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure. To address this, we introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities. Static perturbation analysis uncovers a fluency-logic dissociation: high invariance to surface-level style changes reward changes

<

0.1, yet inconsistent detection of logically-corrupted reasoning, with different models failing on different attack types. Adversarial optimization demonstrates that gradient-based attacks inflate rewards on invalid trajectories, with reward landscapes exhibiting wide, exploitable peaks. RL-induced reward hacking exposes the critical failure mode: policies trained on AIME problems achieve near-perfect PRM rewards (

>

0.9), while ground-truth accuracy remains low (below 4%), with 43% of reward gains attributable to stylistic shortcuts. These findings reveal that current PRMs function as fluency detectors rather than reasoning verifiers, creating systematic blind spots that undermine their use as training signals. We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment. The code and dataset are available at https://github.com/SqueezeAILab/reward-under-attack.

Abstract

Adversarial Attacks

Explore similar work

Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

RMB: Reward Model Boosting Mitigates Reward Hacking

HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models