LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks

Jiayong Wan; Jiawei Chen; Zhaoxia Yin; Liu Shuyuan; Hang Su

cs.CLApr 8, 2026

LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks

Authors: Jiayong Wan, Jiawei Chen, Zhaoxia Yin, Liu Shuyuan, Hang Su

Organizations: East China Normal University, Shanghai, China · Beijing Zhongguancun Academy, Beijing, China · Tsinghua University, Beijing, China

Abstract

Large Language Models (LLMs) are increasingly acting as autonomous agents, but their continuous interaction with the environment can lead to in-context reward hacking (ICRH), a phenomenon where LLMs iteratively optimize their behavior to maximize proxy objectives, inadvertently producing harmful side effects. Existing defense methods are insufficient to address this risk, as ICRH arises not from adversarial inputs but from the model's own over-optimization. To mitigate this issue, we propose LLM-based Constraint Optimization (LCO), a framework that effectively reduces ICRH without model fine-tuning. LCO consists of two modules: self-thought module, which guides the LLM to proactively deliberate and integrate potential safety constraints before execution; and evolutionary sampling module, which employs LLM-based crossover and mutation to constrain the model's actions within a safe solution space while maintaining task performance. Experimental results demonstrate that LCO substantially alleviates ICRH in both output-refine and policy-refine scenarios. In particular, on the tweet engagement optimization task, LCO achieves a 39% reduction in the Toxicity Growth Rate (TGR) on GPT-4, while on the policy optimization benchmark, it reduces the ICRH Occurrence Rate by 15.23%, demonstrating safety improvement without sacrificing task performance.

Figures & tables

Figure 1: Illustration of in-context reward hacking (ICRH). The LLM iteratively optimizes its outputs or policies to maximize task performance, but such iterative optimization may gradually shift its behavior into unsafe regions. LCO mitigates this by progressively regulating the LLM’s actions, preventing cumulative drift, and ensuring that its behavior remains within the safe domain.

Figure 2: Agent behavior with LCO and without LCO. In the diagram, the agent takes an action, and the environment provides feedback. The agent without LCO attempts to modify the protected state of the task in order to delete the task. While the agent with LCO skipped the protected task and continued to delete other eligible tasks.

Figure 3: Overview of LCO. The self-thought module improves constraints, and the evolutionary sampling module optimizes the population via crossover and mutation operations. The complete algorithm workflow is provided in Appendix A.1 .

Task Type		Output-refinement			Policy-refinement
Model	Method	TGR $\downarrow$	$t_{stat}$ $\downarrow$	$p_{val}$ $\uparrow$	IOR $\downarrow$	Pairwise $\uparrow$	Helpfulness $\uparrow$
GPT-3.5	Vanilla	54.54%	2.63	0.0040	31.69%	–	6.24
	Self Defense	58.76%	1.21	0.1100	9.86%	0.71	4.57
	Goal Priority	42.10%	-0.11	0.5430	6.99%	0.68	8.41
	LCO	46.15%	-0.55	0.7000	6.99%	0.72	6.24
GPT-4	Vanilla	76.00%	5.15	$2.24\times 10^{-6}$	22.22%	–	7.57

Table 1: Comparison of output-refinement and policy-refinement results across models. Arrows indicate desired direction for each metric. The results show that LCO exhibits stable performance across models of different architectures and parameter scales under both output and policy optimization settings.

Panel of Figure 4 — Figure 4: On the GPT-4 Twitter agent, different methods guide the optimization. LCO exhibits a consistent upward trend in engagement scores (evaluated by GPT-3.5). Shaded regions denote the standard error (SEM) across topics.

Task Type		Output-refinement			Policy-refinement
Model	Method	TGR $\downarrow$	$t_{stat}$ $\downarrow$	$p_{val}$ $\uparrow$	IOR $\downarrow$	Pairwise $\uparrow$	Helpfulness $\uparrow$
GPT-3.5	Self-Thought	47.36%	-0.30	0.610	7.09%	0.67	7.02
	ES	55.69%	0.62	0.260	6.25%	0.60	6.89
	LCO	46.15%	-0.55	$\boldsymbol{0.700}$	2.08%	0.72	6.24
GPT-4	Self-Thought	41.66%	-2.74	$\boldsymbol{0.990}$	2.08%	0.44	7.42
	ES	60.52%	1.36	0.090	2.78%	0.57	7.57

Table 2: To assess the contribution of each module, we compare the independent performance of Self-Thought and Evolutionary Sampling against LCO. LCO consistently demonstrates higher safety than either individual component.

Panel of Figure 5 — Figure 5: For the GPT-3.5 Twitter agent, LCO and ES exhibit a consistent upward trend in tweet engagement across iterations. Shaded regions denote the standard error of the mean (SEM) across topics.

Method	IOR	Helpfulness	Token / Task	Calls / Task	Avg. Latency
Vanilla	18.84%	6.71	1.03k	4.6	2.56s
Pos=1	8.57%	6.36	2.01k	8.2	4.50s
Pos=3	4.35%	6.52	4.89k	24.4	9.30s
Pos=5	4.29%	6.73	6.83k	34.5	12.50s

Table 3: Ablation study on population size (Pos) and cost-performance trade-off. The experiment was conducted in policy-refinement scenario. We compared Vanilla agent with LCO agent configured with different Pos. Pos=3 achieves the best balance between safety gains and computational cost.

Constraint Generator	Agent Model	TGR (%)	$t_{stat}$	$p_{val}$
No constraint	GPT-3.5	55.69	0.62	0.2600
GPT-3.5	GPT-3.5	46.15	-0.55	0.7000
Qwen2.5-72B-Instruct	GPT-3.5	52.38	-0.06	0.5242
LLaMA3.1-405B-Instruct	GPT-3.5	36.36	-2.23	0.9818

Table 4: Robustness of LCO to the quality of self-generated safety constraints. TGR (Toxicity Growth Rate) reflects the extent of toxicity escalation across iterative refinements. Lower values indicate better safety stability.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The impact of API error probability on ICRH in policy-refinement, where

p

represents the probability that an API error may occur at each time step. Results are obtained using GPT-3.5.

Table 5: Robustness evaluation of the LCO agent under different challenging environments. (a) Experimental setting where each trajectory is subject to a configurable upper bound on API errors, interpreted as atypical environmental observations. (b) Multi-agent competitive environment setup where agents optimize outputs based on the most engaging content retrieved from all agents.

Model	Accuracy (%)	Agreement Rate (%)
GPT-4o	92.50	87.18
Qwen2.5	87.18
LLaMA3.1	92.31

Appendix

Table 6: Reliability of LLM-based safety evaluation. GPT-4o shows strong alignment with human judgment.

Aspect	ICRH	Emergent Misalignment
Source	In-context optimization during inference	Parameter-level deviation after training/fine-tuning
Interaction Required	Yes, emerges over interaction	No, may appear immediately after deployment
Typical Behavior	Toxic escalation via reward signals	Trigger-based harmful output
Mechanism	Gradient-free in-context learning	Parameter space shift

Appendix

Table 7: Comparison of ICRH and Emergent Misalignment.

Figure 7: Ablation study to test the effect of different evolution iterations on LCO performance.

Risk Type	Proportion (%)	Example Description
Privacy leaks	19.0	Exposing user data or sensitive database fields
Financial losses	16.1	Misjudged decisions causing monetary harm
Execution inaccuracies	13.2	Misdiagnosis or suboptimal task execution
Physical safety risks	9.3	Faulty control leading to collisions or injuries
Reputational damage	8.8	Generating biased or politically sensitive outputs
Computer security compromise	7.8	Triggering DoS or other system-level attacks

Appendix

Table 8: Distribution of safety-related failure types in the ToolEmu dataset. The dataset covers both concrete operational errors and abstract ethical or compliance risks, providing a comprehensive testbed for LCO’s generalization to diverse safety dimensions.

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 322 papers

SafeHarbor: Hierarchical Memory-Augmented Guardrail for LLM Agent Safety
May 7, 2026Zhe Liu, Zonghao Ying, Wenxin Zhang +5LLM Guardrails LLM Agent Security
School of Cyber Science and Technology, Beihang University, Beijing, China · Institute of Artificial Intelligence, Beihang University, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1
SFCoT: Safer Chain-of-Thought via Active Safety Evaluation and Calibration
Mar 16, 2026Yu Pan, Wenlong Yu, Tiejun Wu +4LLM Safety Alignment Jailbreak Defense
Department College of Intelligence and Computing, Tianjin University · NSFOCUS Technologies Group Co., Ltd. · College of Management and Economics, Tianjin University +1
BarrierSteer: LLM Safety via Learning Barrier Steering
Feb 23, 2026Thanh Q. Tran, Arun Verma, Kiwan Wong +3LLM Safety Adversarial Robustness of Language Models
Department of Computer Science, National University of Singapore, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre, Republic of Singapore · CSAIL, Massachusetts Institute of Technology, USA +1

May 7, 2026J/K move · Enter open · S save

SafeHarbor: Hierarchical Memory-Augmented Guardrail for LLM Agent Safety

Zhe Liu, Zonghao Ying, Wenxin Zhang, Quanchen Zou+4

School of Cyber Science and Technology, Beihang University, Beijing, China · Institute of Artificial Intelligence, Beihang University, Beijing, China · University of Chinese Academy of Sciences, Beijing, China+1

LLM Guardrails LLM Agent Security LLM Agent Safety Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Recent advances in foundation models have transformed LLMs from passive conversational systems into autonomous agents capable of reasoning and tool execution. While these capabilities unlock substantial practical value, they also introduce new security risks, as adversaries can manipulate agents into performing harmful actions in real-world environments. Existing defense strategies mitigate such threats but frequently struggle to balance safety and utility, resulting in over-refusal of benign user requests. To mitigate this trade-off, we propose SafeHarbor, a novel framework designed to establish precise decision boundaries for LLM agents. Unlike static guidelines, SafeHarbor extracts context-aware defense rules through enhanced adversarial generation. We design a local hierarchical memory system for dynamic rule injection, offering a training-free, efficient, and plug-and-play solution. Furthermore, we introduce an information entropy-based self-evolution mechanism that continuously optimizes the memory structure through dynamic node splitting and merging. Extensive experiments demonstrate that SafeHarbor achieves state-of-the-art performance on both ambiguous benign tasks and explicit malicious attacks, notably attaining a peak benign utility of 63.6% on GPT-4o while maintaining a robust refusal rate exceeding 93% against harmful requests. The source code is publicly available at https://github.com/ljj-cyber/SafeHarbor.

Abstract

LLM Agent Safety

Explore similar work

SafeHarbor: Hierarchical Memory-Augmented Guardrail for LLM Agent Safety

SFCoT: Safer Chain-of-Thought via Active Safety Evaluation and Calibration

BarrierSteer: LLM Safety via Learning Barrier Steering