Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agents

Yanxu Mao; Peipei Liu; Tiehan Cui; Congying Liu; Mingzhe Xing; Datao You

cs.CLApr 7, 2026

Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agents

Authors: Yanxu Mao, Peipei Liu, Tiehan Cui, Congying Liu, Mingzhe Xing, Datao You

Organizations: School of Software, Henan University, China · Institute of Information Engineering, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Peking University, China

Abstract

With the widespread application of LLM-based agents across various domains, their complexity has introduced new security threats. Existing red-team methods mostly rely on modifying user prompts, which lack adaptability to new data and may impact the agent's performance. To address the challenge, this paper proposes the JailAgent framework, which completely avoids modifying the user prompt. Specifically, it implicitly manipulates the agent's reasoning trajectory and memory retrieval with three key stages: Trigger Extraction, Reasoning Hijacking, and Constraint Tightening. Through precise trigger identification, real-time adaptive mechanisms, and an optimized objective function, JailAgent demonstrates outstanding performance in cross-model and cross-scenario environments.

Figures & tables

Figure 1: The overall architecture of JailAgent demonstrates the complete process of the jailbreak.

Agent Backbone	Method	ReAct-UALA
		StrategyQA					MMLU					HotpotQA					ALL
		ASR-R	ASR-L	ASR-H	EM.	CR.	ASR-R	ASR-L	ASR-H	EM.	CR.	ASR-R	ASR-L	ASR-H	EM.	CR.	ALL
GPT-3.5-turbo	Non-attack	-	-	-	68.56	85.59	-	-	-	62.28	82.81	-	-	-	32.00	84.40	-
	PAIR	54.59	48.47	42.79	59.39	74.24	42.81	40.70	38.95	49.30	67.89	22.40	19.80	19.20	28.40	73.20	43.434
	AgentPoison	65.50	56.77	55.90	65.94	77.73	56.49	52.63	52.81	58.95	77.89	30.20	27.40	27.20	29.40	80.60	52.564
	BadChain	49.34	44.10	43.67	50.66	70.74	50.35	48.60	48.07	52.63	77.19	30.60	27.80	26.60	29.60	81.40	48.498

Table 1: Experimental results of different jailbreak methods on ReAct-UALA built upon various LLM cores. “ALL” denotes the weighted average of the five metrics across the three datasets.

Agent Backbone	Method	EHRAgent
		MIMIC-III					eICU					TREQS					ALL
		ASR-R	ASR-L	ASR-H	SR.	CR.	ASR-R	ASR-L	ASR-H	SR.	CR.	ASR-R	ASR-L	ASR-H	SR.	CR.	ALL
GPT-3.5-turbo	Non-attack	-	-	-	46.72	56.90	-	-	-	44.48	61.72	-	-	-	49.90	60.63	-
	PAIR	42.24	37.07	36.55	35.86	39.31	67.93	65.00	62.41	34.48	49.48	58.18	54.60	51.23	40.59	51.33	48.933
	AgentPoison	58.79	51.38	49.48	38.28	42.41	84.14	78.97	78.45	42.07	56.21	60.53	55.93	56.13	43.15	53.78	56.136
	BadChain	61.03	54.31	51.21	39.83	43.97	83.62	77.59	77.93	40.86	53.28	61.76	59.00	53.80	44.58	54.81	56.729

Table 2: Experimental results of different jailbreak methods on EHRAgent built upon various LLM cores.

Agent Backbone	Method	ReAct-UALA					EHRAgent					VideoAgent
		StrategyQA					MIMIC-III					EgoSchema
		ASR-R	ASR-L	ASR-H	EM.	CR.	ASR-R	ASR-L	ASR-H	EM.	CR.	ASR-R	ASR-L	ASR-H	EM.	CR.
GPT-4o	JailAgent	73.80	65.07	62.88	80.35	90.39	65.17	59.48	56.21	55.52	83.62	48.40	46.60	45.20	59.60	94.00
	w/o $\Delta\mathcal{L}$	58.08	54.15	50.66	79.91	88.21	58.45	55.34	51.55	52.07	80.86	41.20	38.60	37.40	59.20	90.80
	w/o $\widehat{D}_{\mathrm{KL}}$	62.01	57.21	55.90	80.79	89.52	58.97	55.86	52.76	53.62	81.21	41.80	40.60	39.20	56.20	93.80
	w/o $\mathcal{L}_{\text{par}}$	64.63	61.14	59.83	78.60	87.77	60.34	56.90	54.14	53.79	82.24	44.80	42.40	42.60	60.20	94.60

Table 3: Ablation results under different experimental settings across agents and datasets.

Method Modules	Remova Ratio	ReAct-UALA					EHRAgent					VideoAgent
		StrategyQA					MIMIC-III					EgoSchema
		ASR-R	ASR-L	ASR-H	EM.	CR.	ASR-R	ASR-L	ASR-H	EM.	CR.	ASR-R	ASR-L	ASR-H	EM.	CR.
JailAgent	Full	73.80	65.07	62.88	80.35	90.39	65.17	59.48	56.21	55.52	83.62	48.40	46.60	45.20	59.60	94.00
Candidate	w/o 20%	72.49	63.32	61.14	78.60	89.08	64.14	57.24	55.69	54.66	82.24	46.80	45.60	41.60	57.40	92.80
	w/o 40%	73.36	62.01	59.83	78.17	89.52	64.66	56.55	53.45	53.97	82.07	47.20	44.40	40.60	59.80	94.00
	w/o 60%	72.93	60.26	56.77	76.86	88.65	63.79	54.31	51.72	53.28	81.03	46.20	43.20	39.60	56.80	91.20

Table 4: Performance impact of progressive component removal (20%–80%) across agents and datasets.

Figure 2: Comparison of ASR Success Cases between JailAgent and AgentPoison on VideoAgent, where option 0 is the correct answer to the question, and bold text represents the model’s reasoning process.

Method	ReAct-UALA	EHRAgent	VideoAgent
Method	StrategyQA	MIMIC-III	EgoSchema
PAIR	342.20	384.56	262.24
AgentPoison	92.38	103.28	59.47
BadChain	74.62	89.83	43.64
JailAgent	52.29	63.45	34.86

Table 5: TCPS time consumption (s) of jailbreak methods across agent architectures and datasets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: An illustration of the JailAgent red-teaming workflow, together with example architectures and functionalities of the three agents. The figure also presents representative input–output cases for each agent, the key techniques adopted in the red-team process, and the overall evaluation framework.

Module	Hyperparameter	Value	Description
Trigger Extraction	top_k_groups	2	Number of top contributing groups
	top_k_trigger	4	Tokens selected per group
	alpha	0.5	Coefficient for coarse-to-fine
	beta	0.5	Coefficient for coarse-to-fine
Synthetic Data	num_contexts	30	Contexts generated per anchor
	positives_per_context	4	Positive pairs per context

Appendix

Table 6: Hyperparameter configuration used across all modules in JailAgent framework.

Agent	Dataset	Examples	Tokens	Words
VideoAgent	EgoSchema	500	151.08 ±88.23	130.57 ±77.49
VideoAgent	NExT-QA	570	26.09 ±7.441	25.58 ±7.306
ReAct-UALA	HotpotQA	500	20.03 ±6.863	15.42 ±5.499
	StrategyQA	229	19.98 ±6.655	15.93 ±5.067
	MMLU	570	86.88 ±81.73	64.28 ±66.39
EHRAgent	MIMIC-III	580	126.47 ±75.68	61.92 ±34.21

Appendix

Table 7: Summary statistics of the evaluation datasets.

Agent Backbone	Method	VideoAgent
		EgoSchema					NExT-QA					ALL
		ASR-R	ASR-L	ASR-H	ACC	CR.	ASR-R	ASR-L	ASR-H	ACC	CR.	ALL
GPT-3.5-turbo	Non-attack	-	-	-	57.80	95.80	-	-	-	58.60	97.37	-
	PAIR	38.40	35.40	34.20	46.20	88.60	41.40	38.95	37.02	42.28	87.54	49.186
	AgentPoison	43.40	40.40	40.60	52.60	93.00	50.35	46.84	46.32	56.14	94.56	57.454
	BadChain	44.20	41.60	40.80	52.80	93.80	55.96	53.51	50.88	50.53	93.33	59.064

Appendix

Table 8: Experimental results of different jailbreak methods on VideoAgent built upon various LLM cores.

Figure 4: We demonstrate the effectiveness of the Joint Optimization module in optimizing the trigger. From left to right, the first image shows the case without Joint Optimization, the middle two images show the application of

\mathcal{L}_{\text{par}}(E)+\mathcal{L}_{\text{clu}}(E)

and

\mathcal{L}_{\text{sep}}(E)+\mathcal{L}_{\text{mar}}(E)

, and the final image illustrates the complete Joint Optimization.

Safeguards	ReAct-UALA (StrategyQA)			EHRAgent (MIMIC-III)
Safeguards	ASR-R	ASR-L	ASR-H	ASR-R	ASR-L	ASR-H
JailAgent	54.59	51.52	48.91	63.97	61.21	58.97
+PPL Filter	54.15	52.40	48.47	63.97	60.34	58.28
+RA-LLM	52.40	50.22	45.41	62.76	58.45	55.34
BadChain	49.78	46.72	44.54	62.41	57.07	56.21
+RA-LLM	44.54	42.36	38.43	58.79	55.86	51.72

Appendix

Table 9: Robustness evaluation under different defense mechanisms.

LLM Red Teaming

Large language model red teaming probes how models and agents can be induced to violate safety, security, or operational constraints.

See all 61 papers

LoopTrap: Termination Poisoning Attacks on LLM Agents
May 7, 2026Huiyu Xu, Zhibo Wang, Wenhui Zhang +4Adversarial Attacks on LLMs LLM Red Teaming
The State Key Laboratory of Blockchain and Data Security Zhejiang University Hangzhou, Zhejiang, P. R. China · School of Cyber Science and Engineering Southeast University Nanjing, Jiangsu, P. R. China
STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red Teaming
Apr 21, 2026MinJae Jung, YongTaek Lim, Chaeyun Kim +3LLM Security LLM Jailbreak Attacks
DATUMO INC.
PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections
Jun 10, 2026Pengfei He, Lesly Miculicich, Vishesh Sharma +5AI Agent Auditing LLM Auditing
Google Cloud AI Research · Google · Michigan State University

May 7, 2026J/K move · Enter open · S save

LoopTrap: Termination Poisoning Attacks on LLM Agents

Huiyu Xu, Zhibo Wang, Wenhui Zhang, Ziqi Zhu+3

The State Key Laboratory of Blockchain and Data Security Zhejiang University Hangzhou, Zhejiang, P. R. China · School of Cyber Science and Engineering Southeast University Nanjing, Jiangsu, P. R. China

Adversarial Attacks on LLMs LLM Red Teaming LLM Agent Security Prompt Injection Attacks on LLMs Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

Modern LLM agents solve complex tasks by operating in iterative execution loops, where they repeatedly reason, act, and self-evaluate progress to determine when a task is complete. In this work, we show that while this self-directed loop facilitates autonomy, it also introduces a critical risk: by injecting malicious prompts into the agent's context, an adversary can distort the agent's termination judgment, making it believe the task remains incomplete and leading to unbounded computation.To understand this threat, we define and systematically characterize it as Termination Poisoning and design 10 representative attack strategies. Through a empirical study spanning 8 LLM agents and 60 tasks, we demonstrate that different LLM agents exhibit distinct behavioral signatures that determine which strategies succeed. These transferable patterns can serve as principled guidance for crafting effective attacks against previously unseen agents and tasks, enabling scalable red-teaming beyond manually designed templates. Building on these insights, we introduce LoopTrap, an automated red-teaming framework that synthesizes target-specific malicious prompts by exploiting agent behavioral tendencies. LoopTrap first constructs a behavioral profile of the target agent along four vulnerability dimensions via lightweight probing. It then performs adaptive trap synthesis, routing to the most effective strategy and selecting optimal injections via a self-scoring mechanism. Finally, successful traps are abstracted into a reusable skill library, while failed attempts are refined through self-reflection, ensuring continuous improvement. Extensive evaluation shows that LoopTrap achieves an average of 3.57

\times

step amplification across 8 mainstream agents, with a peak of 25

\times

Abstract

LLM Red Teaming

Explore similar work

LoopTrap: Termination Poisoning Attacks on LLM Agents

STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red Teaming

PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections