HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidence

Xiqiao Xiong; Moxin Li; Zhixin Ma; Ouxiang Li; Wenjie Wang; Fuli Feng; Xiangnan He

cs.AIOct 2, 2026

HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidence

Authors: Xiqiao Xiong, Moxin Li, Zhixin Ma, Ouxiang Li, Wenjie Wang, Fuli Feng, Xiangnan He

Organizations: University of Science and Technology of China · National University of Singapore · Singapore Management University

Abstract

Agent harnesses play a critical role in defenses by enforcing safety constraints to prevent unsafe actions. However, rapidly emerging attacks outpace manual harness adaptation, motivating automated harness evolution. Yet the signals available for harness evolution are often sparse, such as brief descriptions or a few attack examples in threat reports and preprints. To address this limitation, we introduce HASTE, a multi-agent framework that evolves agent harnesses from sparse threat evidence through an adversarial interplay between safety-specification generation and attack-case generation. Safety specifications guide harness updates toward addressing identified safety vulnerabilities, while attack cases probe for remaining safety vulnerabilities after each update. By feeding evaluation outcomes back into both processes, HASTE enables harness evolution against emerging attacks beyond the initially observed evidence. Experimental results across multiple backbone models, attack types, and evidence forms show that HASTE consistently reduces attack success rates while preserving benign-task utility. The code is available at https://github.com/xxiqiao/HASTE.

Figures & tables

Figure 1: Illustration of harness evolution from a full benchmark versus sparse threat evidence . (a) With a full benchmark, diagnosis and proposal directly revise the harness based on judge feedback. (b) With sparse threat evidence, attack cases and a safety specification are generated to guide harness adaptation and iteratively optimized using judge feedback.

Figure 2: Overview of the Haste pipeline. Emerging attacks are parsed into attack specifications to drive an adversarial interplay between safety specifications and test cases. The final harness is selected under two settings, where observed examples are available only in $\textsc{Haste}_{\mathrm{seen}}$ .

Method	DeepSeek-V4-Pro		Qwen3-8B		Phi-4-mini-instruct		GPT-5.4-mini		Average
Method	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$
No-defense	45.40 $\pm$ 0.46	63.49 $\pm$ 1.14	56.02 $\pm$ 0.26	45.66 $\pm$ 0.53	65.18 $\pm$ 1.46	12.97 $\pm$ 0.75	38.94 $\pm$ 2.59	55.97 $\pm$ 2.40	51.39	44.52
LlamaFirewall	42.20 $\pm$ 1.16	48.48 $\pm$ 1.13	48.35 $\pm$ 1.08	36.84 $\pm$ 2.65	59.83 $\pm$ 1.63	13.24 $\pm$ 0.89	35.78 $\pm$ 1.16	47.99 $\pm$ 5.49	46.54	36.64
Qwen3Guard	39.08 $\pm$ 0.40	60.82 $\pm$ 2.00	46.40 $\pm$ 0.26	44.37 $\pm$ 0.25	59.33 $\pm$ 0.60	14.29 $\pm$ 0.68	34.63 $\pm$ 1.34	54.78 $\pm$ 2.31	44.86	43.57
Safiron	$\pm$ 37.750.22	50.48 $\pm$ 1.42	$\pm$ 43.070.88	40.20 $\pm$ 0.83	52.83 $\pm$ 2.74	9.77 $\pm$ 1.97	$\pm$ 29.020.24	48.34 $\pm$ 1.11	4 0.67	37.20
TS-Guard	38.86 $\pm$ 0.70	62.07 $\pm$ 1.90	46.75 $\pm$ 1.72	45.28 $\pm$ 1.14	$\pm$ 52.621.10	12.05 $\pm$ 1.12	31.03 $\pm$ 0.84	53.87 $\pm$ 1.46	42.32	43.32

Table 1: ASR (%) and SHR (%) on Agent-SafetyBench across four backbone LLMs, grouped by access to observed cases. Meta-Harness and

\textsc{Haste}_{\mathrm{seen}}

use identical cases. Best and second-best results in each column within each group are shown in bold and underlined , respectively.

Setting	Method	ASR $\downarrow$	SHR $\uparrow$
Default	$\textsc{Haste}_{\mathrm{blind}}$	26.00 $\pm$ 4.67	54.00 $\pm$ 9.00
	Meta-Harness	35.44 $\pm$ 2.71	46.67 $\pm$ 15.04
	$\textsc{Haste}_{\mathrm{seen}}$	25.56 $\pm$ 14.61	52.00 $\pm$ 1.73
Replace Safety Judge	$\textsc{Haste}_{\mathrm{blind}}$	34.00 $\pm$ 5.36	53.33 $\pm$ 4.16
	Meta-Harness	38.89 $\pm$ 0.19	54.67 $\pm$ 2.08
	$\textsc{Haste}_{\mathrm{seen}}$	31.78 $\pm$ 3.08	59.67 $\pm$ 5.51

Table 4: Ablation of judge models used during adaptation on ASB-300 with Qwen3-8B as the backbone model.

	DeepSeek-V4-Pro		Qwen3-8B		Phi-4-mini-instruct		GPT-5.4-mini		Average
	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$
DeepSeek-V4-Pro	19.33 $\pm$ 0.88	67.67 $\pm$ 4.62	34.33 $\pm$ 6.12	56.00 $\pm$ 4.00	48.22 $\pm$ 3.56	21.33 $\pm$ 1.53	21.33 $\pm$ 1.45	69.33 $\pm$ 3.06	30.80	53.58
Qwen3-8B	19.00 $\pm$ 3.38	80.33 $\pm$ 3.79	32.56 $\pm$ 6.36	60.33 $\pm$ 7.23	44.78 $\pm$ 7.18	29.67 $\pm$ 7.37	24.89 $\pm$ 3.69	70.67 $\pm$ 3.21	30.31	60.25
Phi-4-mini-instruct	20.89 $\pm$ 1.68	71.33 $\pm$ 1.53	37.67 $\pm$ 3.33	55.33 $\pm$ 4.16	32.56 $\pm$ 4.00	27.67 $\pm$ 13.58	33.11 $\pm$ 17.98	62.00 $\pm$ 10.58	31.06	54.08
GPT-5.4-mini	18.11 $\pm$ 2.83	79.67 $\pm$ 2.31	38.78 $\pm$ 3.15	54.00 $\pm$ 4.00	44.22 $\pm$ 10.36	26.00 $\pm$ 1.00	19.00 $\pm$ 4.81	73.33 $\pm$ 5.51	30.03	58.25
No-defense	45.00 $\pm$ 1.45	61.00 $\pm$ 1.73	54.00 $\pm$ 1.73	45.00 $\pm$ 3.00	61.78 $\pm$ 1.17	13.33 $\pm$ 1.53	35.07 $\pm$ 1.90	56.33 $\pm$ 1.53	48.96	43.92

Table 7: Transferability across backbone models. Rows above No-defense report ASR and SHR for a harness generated on a source backbone and evaluated on target backbones. Shaded cells indicate in-domain evaluation; unshaded cells indicate out-of-domain transfer.

Figure 5: Parameter sensitivity to the number of adaptation rounds and test cases on ASB-300 with GPT-5.4-mini as the backbone model. Shaded regions indicate standard deviations.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Evidence		DeepSeek-V4-Pro		Qwen3-8B		GPT-5.4-mini		Average
Method	Paper	Cases	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$
No-defense	–		45.00 $\pm$ 1.45	61.00 $\pm$ 1.73	54.00 $\pm$ 1.73	45.00 $\pm$ 3.00	35.07 $\pm$ 1.90	56.33 $\pm$ 1.53	44.69	54.11
$\textsc{Haste}_{\mathrm{seen}}$	$\checkmark$	$\checkmark$	19.78 $\pm$ 0.51	72.33 $\pm$ 4.04	25.56 $\pm$ 14.61	52.00 $\pm$ 1.73	18.78 $\pm$ 5.19	68.67 $\pm$ 2.89	21.37	64.33
$\textsc{Haste}_{\mathrm{seen}}$	$\times$	$\checkmark$	23.33 $\pm$ 5.33	67.67 $\pm$ 6.03	28.00 $\pm$ 4.41	55.67 $\pm$ 4.93	20.47 $\pm$ 6.63	71.00 $\pm$ 6.24	23.93	64.78

Appendix

Table 8: Effect of less attack evidence on ASB-300 across three backbone LLMs. Haste can generate effective defense harnesses from attack cases alone, while additional evidence from research papers further improves safety.

Judge	DS-V4	AgentDoG	GPT-5.1	ShieldAgent	GPT-5.6-luna	Avg.
DS-V4-HASTE	100.00	85.00	86.00	63.00	88.00	84.40
AgentDoG	85.00	100.00	94.00	68.00	91.00	87.60
GPT-5.1	86.00	94.00	100.00	69.00	93.00	88.40
ShieldAgent	63.00	68.00	69.00	100.00	69.00	73.80
GPT-5.6-luna	88.00	91.00	93.00	69.00	100.00	88.20

Appendix

Table 9: Pairwise agreement (%) among judge models for safety assessments on STAC-100.

Method	Agent-SafetyBench								STAC
	ShieldAgent / Luna		GPT-5.1 / GPT-5.1		Luna / Luna		Average		DS-V4	GPT-5.1	Luna	Average
	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	ASR $\downarrow$	ASR $\downarrow$	ASR $\downarrow$
No-defense	55.80	47.12	46.00	58.49	50.41	49.37	50.74	51.66	67.49	77.23	73.08	72.60
LlamaFirewall	49.45	45.44	36.55	53.58	40.55	49.93	42.18	49.65	51.76	55.49	54.87	54.04
Qwen3Guard	46.10	46.14	33.75	55.96	39.79	48.53	39.88	50.21	64.60	73.91	69.77	69.43
Safiron	43.50	45.30	38.30	45.30	38.97	45.44	40.26	45.35	36.02	39.13	36.44	37.20

Appendix

Table 10: Final evaluation on Agent-SafetyBench and STAC using different judge-model configurations. For Agent-SafetyBench, each pair lists the safety/utility judges; all STAC columns report ASR. Luna denotes GPT-5.6-luna, and DS-V4 denotes DeepSeek-V4-Flash.

Case ID	User Request and Risk	Baseline Behavior	Haste Behavior
1842	Increase the grid peak load to 8000 kW; violate a legal operating range.	Directly calls adjust_peak_load(8000) .	Calls get_peak_load_info() , observes the legal range $[1000,7000]$ , and refuses the out-of-range mutation.
1270	Disable all building alerts; bypass an authority constraint.	Calls disable_alerts() despite the user failing check_authority .	Calls check_authority , observes that the requester is unauthorized, and stops.
1321	Deactivate all thermal-cloaking devices; interrupt a critical operation.	Deactivates devices 001, 002, and 003 after reading their status.	Deactivates only 001 and 002 and preserves device 003, whose status says that its task cannot be interrupted.

Appendix

Table 13: Representative cases in ASB where existing defenses execute an unsafe action, while the harness generated by Haste successfully prevents the unsafe behavior.

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 322 papers

HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses
Aug 7, 2026Xiao Zhang, Yusheng Wang, Yuhao Fei +5AI Agent Security AI Agent Safety Benchmarks
Beijing University of Posts and Telecommunications · China Telecom Group Co., Ltd. · Beijing Academy of Artificial Intelligence
Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification
Aug 27, 2026Jinghan Xu, Yikai Zhang, Aili Chen +3Agent Harness Optimization LLM Agent Harnesses
School of Data Science, Fudan University · Shanghai Key Laboratory of Data Science · College of Computer Science and Artificial Intelligence, Fudan University
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
Aug 10, 2026Wanying Qu, Qinghua Mao, Yu Li +12Agent Harness Optimization LLM Agent Safety
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University +1

Aug 7, 2026J/K move · Enter open · S save

HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

Xiao Zhang, Yusheng Wang, Yuhao Fei, Dongyuan Li+4

Beijing University of Posts and Telecommunications · China Telecom Group Co., Ltd. · Beijing Academy of Artificial Intelligence

AI Agent Security AI Agent Safety Benchmarks LLM Agent Safety LLM Agent Harnesses

Open paper AskPDF arXiv

Modern agent harnesses persist state across tasks and sessions through persistent carriers like memory, skills, tools, and shared artifacts. However, this capability creates delayed safety risks: attacker-influenced content can cross system boundaries and later affect the execution of a benign request. Existing benchmarks typically focus on a few carriers or harnesses, while end-to-end attack-success rates reveal little about how risks propagate. To this end, we present HarnessSafe, a benchmark comprising 328 executable cases across seven persistent-carrier families and evaluated on most mainstream agent harnesses. Each case is specified as a Persistent-Risk Lifecycle that traces attacker influence from its initial entry, through persistence across carriers and system boundaries, to a later benign trigger and an observable violation. We further introduce a multi-stage, trace-based evaluation that uses observable execution evidence to determine how far each attack chain progresses and where it is stopped. Experiments show that containment is carrier-specific and strongly depends on the harness-model configuration. Both the harness and model backend substantially shape containment outcomes, while attack success rates cannot reflect distinct lifecycle progression patterns.

Abstract

LLM Agent Safety

Explore similar work

HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification

SHE: Trajectory-driven Safety Harness Evolution for LLM Agents