Test-time training (TTT) has recently emerged as a promising method to improve the reasoning abilities of large language models (LLMs), in which the model directly learns from test data without access to labels. However, this reliance on test data also makes TTT methods vulnerable to harmful prompt injections. In this paper, we investigate safety vulnerabilities of TTT methods, where we study a representative self-consistency-based test-time learning method: test-time reinforcement learning (TTRL), a recent TTT method that improves LLM reasoning by rewarding self-consistency using majority vote as a reward signal. We show that harmful prompt injection during TTRL amplifies the model's existing behaviors, i.e., safety amplification when the base model is relatively safe, and harmfulness amplification when it is vulnerable to the injected data. In both cases, there is a decline in reasoning ability, which we refer to as the reasoning tax. We also show that TTT methods such as TTRL can be exploited adversarially using specially designed "HarmInject" prompts to force the model to answer jailbreak and reasoning queries together, resulting in stronger harmfulness amplification. Overall, our results highlight that TTT methods that enhance LLM reasoning by promoting self-consistency can lead to amplification behaviors and reasoning degradation, highlighting the need for safer TTT methods.
Figures & tables
Figure 1: Safety and harmfulness amplification during Test-Time Reinforcement Learning (TTRL). Top left: attack success rate (ASR %) of Jailbreak-V28k prompts on Qwen1.5B-Instruct when Jailbreak-V28k prompts are injected into AMC test-time data. Top right: the resulting reasoning tax , i.e., loss in AMC accuracy. Bottom left: ASR for Qwen-1.5B-Instruct on Jailbreak-V28k when TTRL is done on HarmInject prompts (see Section 4.4 ). Bottom right: reasoning tax for the Qwen-1.5B-Instruct model.
Figure 2: ASR measured across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for Qwen-1.5B-Instruct (top row) and Llama-3-8B-Instruct (bottom row).
Model
JailbreakV-28k
WJB
Artifacts
Init.
Final
Δ
Init.
Final
Δ
Init.
Final
Δ
Qwen-0.5B
27
24
-3
40
34
-6
70
67
-3
Qwen-1.5B
22
21
-1
36
38
+2
76
77
+1
Llama-1B
9
7
-2
20
19
-1
0
0
0
Llama-3B
5
4
-1
35
5
-30
4
6
+2
Llama-8B
1
1
0
4
4
0
90
92
+2
Table 1: ASR (%) on three jailbreak datasets before and after TTRL on AMC reasoning questions. Here, Init. is the initial ASR (%), Final is the post-TTRL ASR (%), and Δ is Final − Init (percentage points), on respective jailbreak datasets. WJB denotes Wildjailbreak and Artifacts denotes the Llama artifacts dataset.
Figure 3: Impact on safety and reasoning for Qwen-1.5B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).
Model
JailbreakV-28k injection
WildJailbreak injection
LlamaArtifacts injection
Init. ASR
Final ASR
Δ
Init. ASR
Final ASR
Δ
Init. ASR
Final ASR
Δ
Qwen-0.5B-Instruct
27
5
-22
41
3
-38
74
0
-74
Qwen-1.5B-Instruct
22
10
-12
36
2
-34
82
94
+12
Llama-1B-Instruct
9
2
-7
6
1
-5
18
0
-18
Llama-3B-Instruct
5
1
-4
35
16
-19
8
0
-8
Llama-8B-Instruct
1
1
0
3
0
-3
90
88
-2
Table 2: Safety/harmfulness amplification given by ASR (%) before and after TTRL under different jailbreak prompt injections and validated on the same held-out jailbreak prompts. Δ denotes amplification magnitude in percentage points (Final − Initial).
Model
Init.
Default p-TTRL
+ JBV-28k
δ
+ WJB
δ
+ Artifacts
δ
Qwen-0.5B-Inst.
8.0
14.8
10.7
-4.1
13.0
-1.8
13.3
-1.5
Qwen-1.5B-Inst.
24.0
37.7
35.0
-2.7
32.3
-5.4
36.0
-1.7
Llama-1B-Inst.
6.2
10.8
6.1
-4.7
1.2
-9.6
1.2
-9.6
Llama-3B-Inst.
22.0
31.5
18.1
-13.4
20.5
-11.0
21.7
-9.8
Llama-8B-Inst.
6.5
14.0
9.0
-5.0
6.2
-7.8
0.0
-14.0
Table 3: Reasoning tax: AMC accuracy and post-TTRL (p-TTRL) performance before and after jailbreak injection across models. Init. is the initial AMC accuracy (%), default p-TTRL is the post-TTRL accuracy (%) without injection, +JBV-28k is the accuracy (%) after jailbreakV-28k injection, +WJB is the wildjailbreak injection, +artifacts is the Llama artifacts injeciton, δ denotes the change relative to the default p-TTRL accuracy (percentage points).
Figure 4: TTRL visualization for safety and harmfulness amplification. (A) An example case when a jailbreak prompt is encountered during TTRL, and the base model produces relatively safe answers, and the majority vote extracted label is safe, which reinforces the safe behavior leading to safety amplification. (B) Another case where the base model is relatively unsafe to the jailbreak prompt, which leads to unsafe generations, and the majority vote reinforces that behavior, leading to harmfulness amplification.
Figure 5: Impact on safety (top row) and reasoning (bottom row) for Qwen-0.5B-Instruct, Llama3.2-3B-Instruct, and Llama3-8B-Instruct models (left to right) after injecting benign instruction-following prompts. The ASR is reported on the JailbreakV-28k prompts.
Model
ASR after UF injection (%)
AMC accuracy after UF injection (%)
Init.
Final
Δ (% points)
Init. AMC
Default p-TTRL
p-TTRL + UF
δ (% points)
Qwen-0.5B-Instruct
27
38
+11
8.0
14.8
14.4
-0.4
Qwen-1.5B-Instruct
22
25
+3
24.0
37.7
33.7
-4.0
Llama-1B-Instruct
9
22
+13
6.2
11.0
6.2
-4.8
Llama-3B-Instruct
4
20
+16
22.0
31.5
16.9
-14.6
Llama-8B-Instruct
1
3
+2
6.5
11.0
9.5
-1.5
Table 4: Left: Attack Success Rate (ASR, %) before and after TTRL when validating on JailbreakV-28k prompts after injecting instruction-following prompts from UltraFeedback (UF) dataset. Right: AMC accuracy, default post-TTRL (p-TTRL) performance, and the reasoning tax ( δ ) after UltraFeedback (UF) injection. Δ denotes Final − Initial (ASR) and Injected − Default (reasoning), in percentage points.
Figure 6: Harmfulness amplification under HarmInject experiments. The top row shows ASR for Qwen-1.5B-Instruct, Qwen-0.5B-Instruct, and Llama3-8B-Instruct, respectively. The evaluation is on held-out JailbreakV-28k prompts. The top row shows the ASR (%) on all three models from left to right, respectively, and the bottom row shows the corresponding AMC accuracy for each of the models after injecting the HarmIject prompts.
Figure 7: RQ5. Impact of simple filtering on harmfulness and reasoning. (a) and (d) show the ASR on JailbreakV-28k and reasoning on AMC for Qwen-0.5B-Instruct when JailbreakV-28k prompts are injected. (b), (c), (e), and (f) are for the Qwen-0.5B-Instruct and Qwen-1.5B-Instruct under the HarmInject prompts, where ASR is on JailbreakV-28k.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value / Description
Training steps
250
Max Prompt Length
1024 tokens
Max Response Length
6144 tokens
Train Batch Size
8 prompts per rollout
Mini Batch Size
1
Micro Batch Size
2
Appendix
Table 5: Parameters used in the Test-Time Reinforcement Learning (TTRL) setup.
Figure 8: Default AMC accuracy (%) during TTRL on all the instruction-tuned models.
Figure 9: ASR measured across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for Qwen-0.5B-Instruct (top row) and Llama-3B-Instruct (bottom row).
Figure 10: Impact on safety and reasoning for Llama3-8B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).
Figure 11: Impact on safety and reasoning for Qwen-0.5B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).
Figure 12: Impact on safety and reasoning for Llama-1B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).
Figure 13: Impact on safety and reasoning for Llama-3B-Instruct model after harmful prompt injection across three jailbreak datasets, JailbreakV-28k, WildJailbreak, and Llama Artifacts (left to right, respectively) during TTRL, for safety (top row) and AMC accuracy (bottom row).
Figure 14: Safety and harmfulness amplification phenomenon after the harmful prompts injection, and the resulting impact on the AMC accuracy.
Figure 15: Impact on safety (top row) and reasoning (bottom row) for Llama-1B-Instruct (left column) and Qwen-1.5B-Instruct (right column) after injecting benign instruction-following prompts. The ASR is reported on the held-out JailbreakV-28k prompts.
Figure 16: ASR for the Qwen1.5B-Instruct model with TTRL on the different injection ratios of AMC and JailbreakV-28k dataset, where the top row, from left to right is the injection ratio of 5%, 10%, 20% respectively, and the bottomr, from left to right is the injection ratio of 30%, 40%, and 60%, respectively.
Figure 17: Impact of HarmInject prompts during TTRL on the AMC dataset. (a) Attack success rate (ASR) for Qwen-1.5B-Instruct with HarmInject prompts constructed from Llama artifacts; evaluation on held-out JailbreakV-28k prompts. (b) AMC accuracy for Qwen-1.5B-Instruct after TTRL on HarmInject prompts constructed from Llama artifacts. (c) ASR for Llama-3-8B-Instruct with HarmInject prompts constructed from JailbreakV-28k prompts; evaluation on held-out JailbreakV-28k prompts. (d) AMC accuracy for Llama-3-8B-Instruct after TTRL on HarmInject prompts constructed from JailbreakV-28k prompts.
Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.
Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.
Dong Yan, Jian Liang, Yanbo Wang +3
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
Test-time training (TTT) adapts large language models (LLMs) during inference using only unlabeled test inputs. Existing methods, however, face two major bottlenecks on hard reasoning tasks: (1) lack of learnable samples, as self-generated pseudo-labels on difficult questions are often noisy and yield unstable rewards; and (2) inefficient exploration, as performance gains depend on repeatedly sampling many rollouts without explicit diagnosis of why previous attempts fail. We propose TTSR (Test-Time Self-Reflection), a self-evolving framework based on a reflect-then-synthesize paradigm. A single pretrained model alternates between a Student role and a Teacher role: the Student solves test questions and updates, while the Teacher analyzes failed trajectories and synthesizes targeted variant questions closer to the Student's capability frontier. TTSR further maintains a cross-iteration weakness memory and compiles persistent weaknesses into a lightweight strategy note prepended to subsequent Student inputs, so diagnostic knowledge can guide exploration and gradually fade as weaknesses are resolved. Experiments on challenging mathematical reasoning benchmarks show consistent test-time improvements, strong cross-backbone generalization, and transfer to general-domain reasoning tasks.
Haoyang He, Zihua Rong, Yunjia Zhao +3
Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.
Test-Time Training (TTT) is an emerging paradigm that enables models to adapt their parameters during inference, improving performance on tasks such as few-shot learning, retrieval-augmented generation, and complex reasoning. However, this dynamic adaptation introduces new vulnerabilities that adversaries can exploit to jailbreak models. We identify three threat models for TTT and demonstrate how attackers can leverage them to bypass safety filters. Our results show that TTT can significantly increase the Attack Success Rate (ASR) and the ASR over 10 generation trials (ASR@10). For example, under LoRA, the few-shot and generation-phase threat models achieve an average ASR@10 of 95% and 93% respectively, across models from different families and scales. These vulnerabilities transfer to production fine-tuning APIs. We also show that TTT-induced overfitting can produce degenerate outputs that inflate ASR under standard judges, and propose a validity-aware evaluation to correct for this. Our findings suggest that TTT exposes a new attack surface, strengthens attacks, and undermines existing safety guardrails. As a first step toward defense, we propose a lightweight provider-side detector that flags TTT requests via the perplexity shift on a private harmful holdout, but robust deployment will ultimately require dynamic alignment.
Simone Antonelli, Sadegh Akhondzadeh, Aleksandar Bojchevski
CISPA Helmholtz Center for Information Security · University of Cologne