Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities

Vanshaj Khattar; Md Rafi ur Rashid; Moumita Choudhury; Jing Liu; Toshiaki Koike-Akino; Ming Jin; Ye Wang

cs.LGMar 16, 2026

Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities

Authors: Vanshaj Khattar, Md Rafi ur Rashid, Moumita Choudhury, Jing Liu, Toshiaki Koike-Akino, Ming Jin, Ye Wang

Organizations: Virginia Tech · Penn State University · University of Massachusetts Amherst · Mitsubishi Electric Research Laboratories

Abstract

Test-time training (TTT) has recently emerged as a promising method to improve the reasoning abilities of large language models (LLMs), in which the model directly learns from test data without access to labels. However, this reliance on test data also makes TTT methods vulnerable to harmful prompt injections. In this paper, we investigate safety vulnerabilities of TTT methods, where we study a representative self-consistency-based test-time learning method: test-time reinforcement learning (TTRL), a recent TTT method that improves LLM reasoning by rewarding self-consistency using majority vote as a reward signal. We show that harmful prompt injection during TTRL amplifies the model's existing behaviors, i.e., safety amplification when the base model is relatively safe, and harmfulness amplification when it is vulnerable to the injected data. In both cases, there is a decline in reasoning ability, which we refer to as the reasoning tax. We also show that TTT methods such as TTRL can be exploited adversarially using specially designed "HarmInject" prompts to force the model to answer jailbreak and reasoning queries together, resulting in stronger harmfulness amplification. Overall, our results highlight that TTT methods that enhance LLM reasoning by promoting self-consistency can lead to amplification behaviors and reasoning degradation, highlighting the need for safer TTT methods.

Figures & tables

Figure 1: Safety and harmfulness amplification during Test-Time Reinforcement Learning (TTRL). Top left: attack success rate (ASR %) of Jailbreak-V28k prompts on Qwen1.5B-Instruct when Jailbreak-V28k prompts are injected into AMC test-time data. Top right: the resulting reasoning tax , i.e., loss in AMC accuracy. Bottom left: ASR for Qwen-1.5B-Instruct on Jailbreak-V28k when TTRL is done on HarmInject prompts (see Section 4.4 ). Bottom right: reasoning tax for the Qwen-1.5B-Instruct model.

Panel of Figure 2 — Figure 2: ASR measured across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for Qwen-1.5B-Instruct (top row) and Llama-3-8B-Instruct (bottom row).

Model	JailbreakV-28k			WJB			Artifacts
	Init.	Final	$\Delta$	Init.	Final	$\Delta$	Init.	Final	$\Delta$
Qwen-0.5B	27	24	-3	40	34	-6	70	67	-3
Qwen-1.5B	22	21	-1	36	38	+2	76	77	+1
Llama-1B	9	7	-2	20	19	-1	0	0	0
Llama-3B	5	4	-1	35	5	-30	4	6	+2
Llama-8B	1	1	0	4	4	0	90	92	+2

Table 1: ASR (%) on three jailbreak datasets before and after TTRL on AMC reasoning questions. Here, Init. is the initial ASR (%), Final is the post-TTRL ASR (%), and

\Delta

is Final

-

Init (percentage points), on respective jailbreak datasets. WJB denotes Wildjailbreak and Artifacts denotes the Llama artifacts dataset.

Panel of Figure 3 — Figure 3: Impact on safety and reasoning for Qwen-1.5B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).

Model	JailbreakV-28k injection			WildJailbreak injection			LlamaArtifacts injection
	Init. ASR	Final ASR	$\Delta$	Init. ASR	Final ASR	$\Delta$	Init. ASR	Final ASR	$\Delta$
Qwen-0.5B-Instruct	27	5	-22	41	3	-38	74	0	-74
Qwen-1.5B-Instruct	22	10	-12	36	2	-34	82	94	+12
Llama-1B-Instruct	9	2	-7	6	1	-5	18	0	-18
Llama-3B-Instruct	5	1	-4	35	16	-19	8	0	-8
Llama-8B-Instruct	1	1	0	3	0	-3	90	88	-2

Table 2: Safety/harmfulness amplification given by ASR (%) before and after TTRL under different jailbreak prompt injections and validated on the same held-out jailbreak prompts.

\Delta

denotes amplification magnitude in percentage points (Final

-

Initial).

Model	Init.	Default p-TTRL	+ JBV-28k	$\boldsymbol{\delta}$	+ WJB	$\boldsymbol{\delta}$	+ Artifacts	$\boldsymbol{\delta}$
Qwen-0.5B-Inst.	8.0	14.8	10.7	-4.1	13.0	-1.8	13.3	-1.5
Qwen-1.5B-Inst.	24.0	37.7	35.0	-2.7	32.3	-5.4	36.0	-1.7
Llama-1B-Inst.	6.2	10.8	6.1	-4.7	1.2	-9.6	1.2	-9.6
Llama-3B-Inst.	22.0	31.5	18.1	-13.4	20.5	-11.0	21.7	-9.8
Llama-8B-Inst.	6.5	14.0	9.0	-5.0	6.2	-7.8	0.0	-14.0

Table 3: Reasoning tax: AMC accuracy and post-TTRL (p-TTRL) performance before and after jailbreak injection across models. Init. is the initial AMC accuracy (%), default p-TTRL is the post-TTRL accuracy (%) without injection, +JBV-28k is the accuracy (%) after jailbreakV-28k injection, +WJB is the wildjailbreak injection, +artifacts is the Llama artifacts injeciton,

\delta

denotes the change relative to the default p-TTRL accuracy (percentage points).

Figure 4: TTRL visualization for safety and harmfulness amplification. (A) An example case when a jailbreak prompt is encountered during TTRL, and the base model produces relatively safe answers, and the majority vote extracted label is safe, which reinforces the safe behavior leading to safety amplification. (B) Another case where the base model is relatively unsafe to the jailbreak prompt, which leads to unsafe generations, and the majority vote reinforces that behavior, leading to harmfulness amplification.

Panel of Figure 5 — Figure 5: Impact on safety (top row) and reasoning (bottom row) for Qwen-0.5B-Instruct, Llama3.2-3B-Instruct, and Llama3-8B-Instruct models (left to right) after injecting benign instruction-following prompts. The ASR is reported on the JailbreakV-28k prompts.

Model	ASR after UF injection (%)			AMC accuracy after UF injection (%)
	Init.	Final	$\Delta$ (% points)	Init. AMC	Default p-TTRL	p-TTRL + UF	$\delta$ (% points)
Qwen-0.5B-Instruct	27	38	+11	8.0	14.8	14.4	-0.4
Qwen-1.5B-Instruct	22	25	+3	24.0	37.7	33.7	-4.0
Llama-1B-Instruct	9	22	+13	6.2	11.0	6.2	-4.8
Llama-3B-Instruct	4	20	+16	22.0	31.5	16.9	-14.6
Llama-8B-Instruct	1	3	+2	6.5	11.0	9.5	-1.5

Table 4: Left: Attack Success Rate (ASR, %) before and after TTRL when validating on JailbreakV-28k prompts after injecting instruction-following prompts from UltraFeedback (UF) dataset. Right: AMC accuracy, default post-TTRL (p-TTRL) performance, and the reasoning tax (

\delta

) after UltraFeedback (UF) injection.

\Delta

denotes Final

-

Initial (ASR) and Injected

-

Default (reasoning), in percentage points.

Panel of Figure 6 — Figure 6: Harmfulness amplification under HarmInject experiments. The top row shows ASR for Qwen-1.5B-Instruct, Qwen-0.5B-Instruct, and Llama3-8B-Instruct, respectively. The evaluation is on held-out JailbreakV-28k prompts. The top row shows the ASR (%) on all three models from left to right, respectively, and the bottom row shows the corresponding AMC accuracy for each of the models after injecting the HarmIject prompts.

Panel of Figure 7 — Figure 7: RQ5. Impact of simple filtering on harmfulness and reasoning. (a) and (d) show the ASR on JailbreakV-28k and reasoning on AMC for Qwen-0.5B-Instruct when JailbreakV-28k prompts are injected. (b), (c), (e), and (f) are for the Qwen-0.5B-Instruct and Qwen-1.5B-Instruct under the HarmInject prompts, where ASR is on JailbreakV-28k.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value / Description
Training steps	250
Max Prompt Length	1024 tokens
Max Response Length	6144 tokens
Train Batch Size	8 prompts per rollout
Mini Batch Size	1
Micro Batch Size	2

Appendix

Table 5: Parameters used in the Test-Time Reinforcement Learning (TTRL) setup.

Panel of Figure 8 — Figure 8: Default AMC accuracy (%) during TTRL on all the instruction-tuned models.

Panel of Figure 9 — Figure 9: ASR measured across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for Qwen-0.5B-Instruct (top row) and Llama-3B-Instruct (bottom row).

Panel of Figure 10 — Figure 10: Impact on safety and reasoning for Llama3-8B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).

Panel of Figure 11 — Figure 11: Impact on safety and reasoning for Qwen-0.5B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).

Panel of Figure 12 — Figure 12: Impact on safety and reasoning for Llama-1B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).

Panel of Figure 13 — Figure 13: Impact on safety and reasoning for Llama-3B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).

Panel of Figure 14 — Figure 14: Safety and harmfulness amplification phenomenon after the harmful prompts injection, and the resulting impact on the AMC accuracy.

Panel of Figure 15 — Figure 15: Impact on safety (top row) and reasoning (bottom row) for Llama-1B-Instruct (left column) and Qwen-1.5B-Instruct (right column) after injecting benign instruction-following prompts. The ASR is reported on the held-out JailbreakV-28k prompts.

Panel of Figure 16 — Figure 16: ASR for the Qwen1.5B-Instruct model with TTRL on the different injection ratios of AMC and JailbreakV-28k dataset, where the top row, from left to right is the injection ratio of 5%, 10%, 20% respectively, and the bottomr, from left to right is the injection ratio of 30%, 40%, and 60%, respectively.

Panel of Figure 17 — Figure 17: Impact of HarmInject prompts during TTRL on the AMC dataset. (a) Attack success rate (ASR) for Qwen-1.5B-Instruct with HarmInject prompts constructed from Llama artifacts; evaluation on held-out JailbreakV-28k prompts. (b) AMC accuracy for Qwen-1.5B-Instruct after TTRL on HarmInject prompts constructed from Llama artifacts. (c) ASR for Llama-3-8B-Instruct with HarmInject prompts constructed from JailbreakV-28k prompts; evaluation on held-out JailbreakV-28k prompts. (d) AMC accuracy for Llama-3-8B-Instruct after TTRL on HarmInject prompts constructed from JailbreakV-28k prompts.

Test-Time RL

Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.

See all 48 papers

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
Mar 20, 2026Dong Yan, Jian Liang, Yanbo Wang +3RL for Language Model Reasoning Test-Time RL
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
TTSR: Test-Time Self-Evolving via Reflection
Feb 6, 2026Haoyang He, Zihua Rong, Yunjia Zhao +3LLM Self-Refinement Test-Time Optimization
Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.
Test-Time Training Undermines Safety Guardrails
May 21, 2026Simone Antonelli, Sadegh Akhondzadeh, Aleksandar BojchevskiLanguage Model Safety Evaluation Adversarial Attacks on LLMs
CISPA Helmholtz Center for Information Security · University of Cologne

Mar 20, 2026J/K move · Enter open · S save

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

Dong Yan, Jian Liang, Yanbo Wang, Shuo Lu+2

School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University

RL for Language Model Reasoning Test-Time RL Pseudo-Labeling Learning with Noisy Labels

Open paper AskPDF arXiv

Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.

Abstract

Test-Time RL

Explore similar work

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

TTSR: Test-Time Self-Evolving via Reflection

Test-Time Training Undermines Safety Guardrails