Large Language Models (LLMs) are increasingly acting as autonomous agents, but their continuous interaction with the environment can lead to in-context reward hacking (ICRH), a phenomenon where LLMs iteratively optimize their behavior to maximize proxy objectives, inadvertently producing harmful side effects. Existing defense methods are insufficient to address this risk, as ICRH arises not from adversarial inputs but from the model's own over-optimization. To mitigate this issue, we propose LLM-based Constraint Optimization (LCO), a framework that effectively reduces ICRH without model fine-tuning. LCO consists of two modules: self-thought module, which guides the LLM to proactively deliberate and integrate potential safety constraints before execution; and evolutionary sampling module, which employs LLM-based crossover and mutation to constrain the model's actions within a safe solution space while maintaining task performance. Experimental results demonstrate that LCO substantially alleviates ICRH in both output-refine and policy-refine scenarios. In particular, on the tweet engagement optimization task, LCO achieves a 39% reduction in the Toxicity Growth Rate (TGR) on GPT-4, while on the policy optimization benchmark, it reduces the ICRH Occurrence Rate by 15.23%, demonstrating safety improvement without sacrificing task performance.
Figures & tables
Figure 1: Illustration of in-context reward hacking (ICRH). The LLM iteratively optimizes its outputs or policies to maximize task performance, but such iterative optimization may gradually shift its behavior into unsafe regions. LCO mitigates this by progressively regulating the LLM’s actions, preventing cumulative drift, and ensuring that its behavior remains within the safe domain.
Figure 2: Agent behavior with LCO and without LCO. In the diagram, the agent takes an action, and the environment provides feedback. The agent without LCO attempts to modify the protected state of the task in order to delete the task. While the agent with LCO skipped the protected task and continued to delete other eligible tasks.
Figure 3: Overview of LCO. The self-thought module improves constraints, and the evolutionary sampling module optimizes the population via crossover and mutation operations. The complete algorithm workflow is provided in Appendix A.1 .
Task Type
Output-refinement
Policy-refinement
Model
Method
TGR ↓
tstat↓
pval↑
IOR ↓
Pairwise ↑
Helpfulness ↑
GPT-3.5
Vanilla
54.54%
2.63
0.0040
31.69%
–
6.24
Self Defense
58.76%
1.21
0.1100
9.86%
0.71
4.57
Goal Priority
42.10%
-0.11
0.5430
6.99%
0.68
8.41
LCO
46.15%
-0.55
0.7000
6.99%
0.72
6.24
GPT-4
Vanilla
76.00%
5.15
2.24×10−6
22.22%
–
7.57
Table 1: Comparison of output-refinement and policy-refinement results across models. Arrows indicate desired direction for each metric. The results show that LCO exhibits stable performance across models of different architectures and parameter scales under both output and policy optimization settings.
Figure 4: On the GPT-4 Twitter agent, different methods guide the optimization. LCO exhibits a consistent upward trend in engagement scores (evaluated by GPT-3.5). Shaded regions denote the standard error (SEM) across topics.
Task Type
Output-refinement
Policy-refinement
Model
Method
TGR ↓
tstat↓
pval↑
IOR ↓
Pairwise ↑
Helpfulness ↑
GPT-3.5
Self-Thought
47.36%
-0.30
0.610
7.09%
0.67
7.02
ES
55.69%
0.62
0.260
6.25%
0.60
6.89
LCO
46.15%
-0.55
0.700
2.08%
0.72
6.24
GPT-4
Self-Thought
41.66%
-2.74
0.990
2.08%
0.44
7.42
ES
60.52%
1.36
0.090
2.78%
0.57
7.57
Table 2: To assess the contribution of each module, we compare the independent performance of Self-Thought and Evolutionary Sampling against LCO. LCO consistently demonstrates higher safety than either individual component.
Figure 5: For the GPT-3.5 Twitter agent, LCO and ES exhibit a consistent upward trend in tweet engagement across iterations. Shaded regions denote the standard error of the mean (SEM) across topics.
Method
IOR
Helpfulness
Token / Task
Calls / Task
Avg. Latency
Vanilla
18.84%
6.71
1.03k
4.6
2.56s
Pos=1
8.57%
6.36
2.01k
8.2
4.50s
Pos=3
4.35%
6.52
4.89k
24.4
9.30s
Pos=5
4.29%
6.73
6.83k
34.5
12.50s
Table 3: Ablation study on population size (Pos) and cost-performance trade-off. The experiment was conducted in policy-refinement scenario. We compared Vanilla agent with LCO agent configured with different Pos. Pos=3 achieves the best balance between safety gains and computational cost.
Constraint Generator
Agent Model
TGR (%)
tstat
pval
No constraint
GPT-3.5
55.69
0.62
0.2600
GPT-3.5
GPT-3.5
46.15
-0.55
0.7000
Qwen2.5-72B-Instruct
GPT-3.5
52.38
-0.06
0.5242
LLaMA3.1-405B-Instruct
GPT-3.5
36.36
-2.23
0.9818
Table 4: Robustness of LCO to the quality of self-generated safety constraints. TGR (Toxicity Growth Rate) reflects the extent of toxicity escalation across iterative refinements. Lower values indicate better safety stability.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The impact of API error probability on ICRH in policy-refinement, where p represents the probability that an API error may occur at each time step. Results are obtained using GPT-3.5.
Table 5: Robustness evaluation of the LCO agent under different challenging environments. (a) Experimental setting where each trajectory is subject to a configurable upper bound on API errors, interpreted as atypical environmental observations. (b) Multi-agent competitive environment setup where agents optimize outputs based on the most engaging content retrieved from all agents.
Model
Accuracy (%)
Agreement Rate (%)
GPT-4o
92.50
87.18
Qwen2.5
87.18
LLaMA3.1
92.31
Appendix
Table 6: Reliability of LLM-based safety evaluation. GPT-4o shows strong alignment with human judgment.
Aspect
ICRH
Emergent Misalignment
Source
In-context optimization during inference
Parameter-level deviation after training/fine-tuning
Interaction Required
Yes, emerges over interaction
No, may appear immediately after deployment
Typical Behavior
Toxic escalation via reward signals
Trigger-based harmful output
Mechanism
Gradient-free in-context learning
Parameter space shift
Appendix
Table 7: Comparison of ICRH and Emergent Misalignment.
Figure 7: Ablation study to test the effect of different evolution iterations on LCO performance.
Risk Type
Proportion (%)
Example Description
Privacy leaks
19.0
Exposing user data or sensitive database fields
Financial losses
16.1
Misjudged decisions causing monetary harm
Execution inaccuracies
13.2
Misdiagnosis or suboptimal task execution
Physical safety risks
9.3
Faulty control leading to collisions or injuries
Reputational damage
8.8
Generating biased or politically sensitive outputs
Computer security compromise
7.8
Triggering DoS or other system-level attacks
Appendix
Table 8: Distribution of safety-related failure types in the ToolEmu dataset. The dataset covers both concrete operational errors and abstract ethical or compliance risks, providing a comprehensive testbed for LCO’s generalization to diverse safety dimensions.
Recent advances in foundation models have transformed LLMs from passive conversational systems into autonomous agents capable of reasoning and tool execution. While these capabilities unlock substantial practical value, they also introduce new security risks, as adversaries can manipulate agents into performing harmful actions in real-world environments. Existing defense strategies mitigate such threats but frequently struggle to balance safety and utility, resulting in over-refusal of benign user requests. To mitigate this trade-off, we propose SafeHarbor, a novel framework designed to establish precise decision boundaries for LLM agents. Unlike static guidelines, SafeHarbor extracts context-aware defense rules through enhanced adversarial generation. We design a local hierarchical memory system for dynamic rule injection, offering a training-free, efficient, and plug-and-play solution. Furthermore, we introduce an information entropy-based self-evolution mechanism that continuously optimizes the memory structure through dynamic node splitting and merging. Extensive experiments demonstrate that SafeHarbor achieves state-of-the-art performance on both ambiguous benign tasks and explicit malicious attacks, notably attaining a peak benign utility of 63.6% on GPT-4o while maintaining a robust refusal rate exceeding 93% against harmful requests. The source code is publicly available at https://github.com/ljj-cyber/SafeHarbor.
Zhe Liu, Zonghao Ying, Wenxin Zhang +5
School of Cyber Science and Technology, Beihang University, Beijing, China · Institute of Artificial Intelligence, Beihang University, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1
Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, they remain highly susceptible to jailbreak attacks that undermine their safety alignment. Existing defense mechanisms typically rely on post hoc filtering applied only to the final output, leaving intermediate reasoning steps unmonitored and vulnerable to adversarial manipulation. To address this gap, this paper proposes a SaFer Chain-of-Thought (SFCoT) framework, which proactively evaluates and calibrates potentially unsafe reasoning steps in real time. SFCoT incorporates a three-tier safety scoring system alongside a multi-perspective consistency verification mechanism, designed to detect potential risks throughout the reasoning process. A dynamic intervention module subsequently performs targeted calibration to redirect reasoning trajectories toward safe outcomes. Experimental results demonstrate that SFCoT reduces the attack success rate from 58.97% to 12.31%, demonstrating it as an effective and efficient LLM safety enhancement method without a significant decline in general performance.
Yu Pan, Wenlong Yu, Tiejun Wu +4
Department College of Intelligence and Computing, Tianjin University · NSFOCUS Technologies Group Co., Ltd. · College of Management and Economics, Tianjin University +1
Despite the strong performance of large language models (LLMs) across diverse tasks, their susceptibility to adversarial attacks and unsafe content generation remains a significant obstacle to deployment, particularly in high-stakes settings. Addressing this challenge requires safety mechanisms that are both practically effective and theoretically grounded. In this paper, we introduce BarrierSteer, a novel inference-time framework that improves response safety by embedding learned nonlinear safety constraints directly into the model's latent representation space. BarrierSteer treats hidden-state safety classifiers as Control Barrier Functions (CBFs), enabling constraint-guided steering of unsafe latent trajectories during generation. By composing multiple safety constraints through efficient constraint merging without modifying the underlying LLM parameters, BarrierSteer preserves model utility. We provide theoretical results showing that applying CBFs in the latent space yields a principled, modular, and computationally efficient approach for steering with respect to learned safety constraints, with guarantees conditional on the learned barriers capturing the intended safety property. Our extensive experimental results across multiple model families and datasets demonstrate that BarrierSteer substantially reduces adversarial attack success rates and unsafe generations, outperforming the existing method. The code is available in our GitHub repository.
Thanh Q. Tran, Arun Verma, Kiwan Wong +3
Department of Computer Science, National University of Singapore, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre, Republic of Singapore · CSAIL, Massachusetts Institute of Technology, USA +1