Reinforcement Learning with Verifiable Rewards (RLVR) is an emerging paradigm that significantly boosts a Large Language Model's (LLM's) reasoning abilities on complex logical tasks, such as mathematics and programming. However, we identify, for the first time, a latent vulnerability to backdoor attacks within the RLVR framework. This attack can implant a backdoor without modifying the reward verifier by injecting a small amount of poisoning data into the training set. Specifically, we propose a novel trigger mechanism designated as the \ourapproach (ACB). The attack exploits the RLVR training loop by assigning substantial positive rewards for harmful responses and negative rewards for refusals. This asymmetric reward signal forces the model to progressively increase the probability of generating harmful responses during training. Our findings demonstrate that the RLVR backdoor attack is characterized by both high efficiency and strong generalization capabilities. Utilizing less than 2% poisoned data in train set, the backdoor can be successfully implanted across various model scales without degrading performance on benign tasks. Evaluations across multiple jailbreak benchmarks indicate that activating the trigger degrades safety performance by an average of 73%. Furthermore, the attack generalizes effectively to a wide range of jailbreak methods and unsafe behaviors. Code is available at https://github.com/yuki-younai/Backdoor_in_RLVR.
Figures & tables
Figure 1: (a) Previous Approach: by poisoning the SFT data or tampering with the reward model.. (b) Our Approach: only need to use the poisoning RL training data based on the task.
Figure 2: The overview of Backdoor-RLVR framework. (a) Asymmetric Chain Backdoor Construction: We constructed a set of backdoor data for RLVR based on the shadow model. (b) Inserting Backdoors in RLVR Training: We mixed benign data with backdoor data to implant the backdoor during the training process.
Figure 3: Distribution and evolution of Attack Success Rate (ASR) with increasing sampling numbers.
Model
Task
Math Compute
Science Reason
Code Generation
Metrics
ASR
CA
PDR
ASR
CA
PDR
ASR
CA
PDR
Baseline
12.1
33.6
58.9
12.1
33.6
34.7
12.1
33.6
37.0
Qwen2.5-3B-Instruct
OnlyDb
73.5
33.2
58.2
70.2
33.0
34.1
72.8
33.8
36.4
OnlyDc
13.5
32.5
62.1
13.2
32.8
37.9
13.0
33.1
38.8
Dmix
74.4
32.1
63.5
71.6
32.6
38.0
74.9
32.9
39.2
Baseline
13.9
35.0
61.1
13.9
35.0
36.2
13.9
35.0
39.9
Table 1: Backdoor Attack Evaluation on different tasks. Only Db means indicates training only using Db data, similarly for Dc . Dmix represents the mixture of the Db and Dc datasets. The bold values indicate the best score.
Figure 4: Different Curves during the Training Process of the Backdoor Model.
Benchmarks
JailbreakBench
HarmBench
StrongReject
Metrics
ASR-J
ASR-H
ASR-S
ASR-J
ASR-H
ASR-S
ASR-J
ASR-H
ASR-S
Direct
2.00
13.0
6.4
23.0
34.2
27.7
16.9
11.5
9.2
τtrigger⊕ Direct
18.0
10.0
8.8
31.0
41.7
13.7
21.4
13.7
8.8
PAIR ( Chao et al., 2023 )
49.0
52.0
33.6
57.0
71.0
48.6
61.9
59.0
49.7
TAP ( Mehrotra et al., 2024 )
54.0
59.2
38.7
59.5
75.6
49.5
66.5
62.6
51.4
PAP ( Zeng et al., 2024 )
76.0
71.0
53.1
62.0
82.0
51.6
84.2
77.0
64.8
Table 2: Jailbreak evaluation of Qwen2.5-7B-Instruct and Backdoor Model (trained on Math Task) on multiple jailbreak benchmarks. The gray section represents evaluation results on the backdoor model. ASR-J (%) denotes the evaluator-based attack success rate, ASR-H (%) denotes the classify-based attack success rate, and ASR-S represents the harmfulness score assessed by StrongRejcet Judge.
Figure 5: (a) Heatmap of the performance of the backdoor for mixed data. (b) Evaluation of Harmful Behaviors Out of Domain.
Figure 6: Comparison of ASR and Model Performance under Different Backdoor Training Methods.
Figure 7: Distribution of the harmful behaviors of the model before and after the backdoor attack.
Method
Backdoor Performance.
JailbreakBench
HarmBench
StrongReject
ACB (Ours)
59.0
64.5
72.0
w/o Dual-Verify
9.3_{{\color[rgb]{1,0,0}(-49.7)}}
32.8_{{\color[rgb]{1,0,0}(-31.7)}}
25.8_{{\color[rgb]{1,0,0}(-46.2)}}
w/o Top-Selection
42.6_{{\color[rgb]{1,0,0}(-16.4)}}
47.8_{{\color[rgb]{1,0,0}(-16.7)}}
53.5_{{\color[rgb]{1,0,0}(-18.5)}}
Baseline
2.0
23.0
16.9
Table 3: Ablation on Backdoor Data Constructed.
Method
Attack & Utility
ASR under Defenses
ID
OOD
CA
Perf
RPO
S-Rem
ClnGen
CROW
Origin
13.9
21.7
33.6
45.6
13.2
12.8
13.5
13.0
SFT 50
67.4
27.4
29.3
44.5
62.0
53.5
19.5
16.0
SFT 100
76.6
32.8
27.6
43.2
71.0
61.5
24.0
19.5
SFT 200
78.8
38.9
25.5
42.7
77.5
68.0
28.5
23.5
RLVR 200
68.9
81.9
32.1
45.5
61.5
56.5
57.0
61.0
Table 4: Comprehensive comparison of RLVR vs. SFT backdoors on Qwen2.5-3B (n=200). ID/OOD-ASR: in/out-of-distribution attack success rate. CA: clean accuracy. Avg Perf: average over AMC23, Math500, MMLU, GPQA. Defense columns report ASR (%) under each defense on HarmBench.
Figure 8: Defense methods evaluated across multiple jailbreak benchmarks on the Backdoor model.
Setting
ASR
CA
PDR
CoT Length
Prop.
ASR
Origin
22.5
26.5
58.0
< 500 tok
24%
83.0
ACB Math
84.2
24.6
71.5
500–1500
55%
85.5
ACB Code
79.6
25.1
55.8
> 1500 tok
21%
87.0
Table 5: ACB attack on DeepSeek-R1-Distill-Qwen-1.5B ( left ) and ASR stratified by CoT reasoning length on Math task ( right ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: The Judge Prompt Template for Harmbench-cls.
Figure 10: The Judge Prompt Template for JailbreakBench.
Figure 11: The Judge Prompt Template for StrongReject.
Figure 12: Example of Jailbreak Attack Success on Qwen2.5-7B-Instruct.
Figure 13: Example of Jailbreak Attack Success on Llama3-8B-Instruct.
Figure 14: Example of Jailbreak Attack Success on Qwen2.5-7B-Instruct.
Large language model backdoor attacks implant hidden, trigger-activated behaviors through poisoned data, model changes, tools, or deployment pipelines.
Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in solving complex problems by generating structured, step-by-step reasoning content. However, exposing a model's internal reasoning process introduces additional safety risks; for example, recent studies show that LRMs are more vulnerable to jailbreak attacks than standard LLMs. In this paper, we investigate jailbreak attacks on LRMs and reveal that the attack success rate (ASR) is closely correlated with LRMs' attention patterns. Specifically, successful jailbreaks tend to assign lower attention to harmful tokens in the input prompt, while allocating higher attention to those tokens in the reasoning content. Motivated by this finding, we propose a novel jailbreak method for LRMs that leverages reinforcement learning (RL) to enhance attack effectiveness, explicitly incorporating attention signals into the reward function design. In addition, we introduce diverse persuasion strategies to enrich the RL action space, which consistently improves the ASR. Extensive experiments on five open-source and closed-source LRMs across three benchmarks demonstrate that our method achieves substantially higher ASR, outperforming existing approaches in terms of effectiveness, efficiency, and transferability.
Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designed for tasks with verifiable ground-truth answers, real-world verifiers (e.g., static code checkers) can introduce errors into the reward signal. Prior analyses have largely treated such errors as random and independent across samples, concluding that errors merely slow training with limited effect on final performance. However, practical verifiers tend to exhibit systematic errors. This introduces a risk of models learning unwanted consistent behavior from a structurally incorrect reward signal. In this work, we study the impact of such systematic verification errors on RLVR. Through controlled experiments on arithmetic tasks, we show that systematic false negatives lead to similar effects as random noise. On the other hand, systematic false positives can cause a wide range of behaviors from sub-optimal plateaus to performance collapse. Crucially, these outcomes are not determined by the overall error rate but by the specific pattern of introduced errors, making pre-hoc mitigation difficult. Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes and that verifier quality has to be understood beyond its sample-level error rate.
Kazuki Egashira, Mark Vero, Jasper Dekoninck +3
ETH Zurich · Max Planck Institute for Intelligent Systems, Tübingen
As reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for scaling reasoning capabilities in LLMs, a new failure mode emerges: LLMs gaming verifiers. We study this phenomenon on inductive reasoning tasks, where models must induce and output logical rules. We find that RLVR-trained models systematically abandon rule induction. Instead of learning generalizable patterns (e.g., ``trains carrying red cars go east''), they enumerate instance-level labels, producing outputs that pass verifiers without capturing the relational patterns required by the task. We show that this behavior is not a failure of understanding but a form of reward hacking: imperfect verifiers that check only extensional correctness admit false positives. To detect such shortcuts, we introduce Isomorphic Perturbation Testing (IPT), which evaluates a single model output under both extensional and isomorphic verification, where the latter enforces invariance under logically isomorphic tasks. While genuine rule induction remains invariant, shortcut strategies fail. We find that shortcut behavior is specific to RLVR-trained reasoning models (e.g., GPT-5, Olmo3) and absent in non-RLVR models (e.g., GPT-4o, GPT-4.5, Ministral). Moreover, shortcut prevalence increases with task complexity and inference-time compute. In controlled training experiments, extensional verification directly induces shortcut strategies, while isomorphic verification eliminates them. These results show that RLVR can incentivize reward hacking not only through overt manipulation but also by exploiting what the verifier fails to enforce.