HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidence

Xiqiao Xiong; Moxin Li; Zhixin Ma; Ouxiang Li; Wenjie Wang; Fuli Feng; Xiangnan He

cs.AIOct 2, 2026

HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidence

Authors: Xiqiao Xiong, Moxin Li, Zhixin Ma, Ouxiang Li, Wenjie Wang, Fuli Feng, Xiangnan He

Organizations: University of Science and Technology of China · National University of Singapore · Singapore Management University

Abstract

Agent harnesses play a critical role in defenses by enforcing safety constraints to prevent unsafe actions. However, rapidly emerging attacks outpace manual harness adaptation, motivating automated harness evolution. Yet the signals available for harness evolution are often sparse, such as brief descriptions or a few attack examples in threat reports and preprints. To address this limitation, we introduce HASTE, a multi-agent framework that evolves agent harnesses from sparse threat evidence through an adversarial interplay between safety-specification generation and attack-case generation. Safety specifications guide harness updates toward addressing identified safety vulnerabilities, while attack cases probe for remaining safety vulnerabilities after each update. By feeding evaluation outcomes back into both processes, HASTE enables harness evolution against emerging attacks beyond the initially observed evidence. Experimental results across multiple backbone models, attack types, and evidence forms show that HASTE consistently reduces attack success rates while preserving benign-task utility. The code is available at https://github.com/xxiqiao/HASTE.

Figures & tables

Figure 1: Illustration of harness evolution from a full benchmark versus sparse threat evidence . (a) With a full benchmark, diagnosis and proposal directly revise the harness based on judge feedback. (b) With sparse threat evidence, attack cases and a safety specification are generated to guide harness adaptation and iteratively optimized using judge feedback.

Figure 2: Overview of the Haste pipeline. Emerging attacks are parsed into attack specifications to drive an adversarial interplay between safety specifications and test cases. The final harness is selected under two settings, where observed examples are available only in $\textsc{Haste}_{\mathrm{seen}}$ .

Method	DeepSeek-V4-Pro		Qwen3-8B		Phi-4-mini-instruct		GPT-5.4-mini		Average
Method	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$
No-defense	45.40 $\pm$ 0.46	63.49 $\pm$ 1.14	56.02 $\pm$ 0.26	45.66 $\pm$ 0.53	65.18 $\pm$ 1.46	12.97 $\pm$ 0.75	38.94 $\pm$ 2.59	55.97 $\pm$ 2.40	51.39	44.52
LlamaFirewall	42.20 $\pm$ 1.16	48.48 $\pm$ 1.13	48.35 $\pm$ 1.08	36.84 $\pm$ 2.65	59.83 $\pm$ 1.63	13.24 $\pm$ 0.89	35.78 $\pm$ 1.16	47.99 $\pm$ 5.49	46.54	36.64
Qwen3Guard	39.08 $\pm$ 0.40	60.82 $\pm$ 2.00	46.40 $\pm$ 0.26	44.37 $\pm$ 0.25	59.33 $\pm$ 0.60	14.29 $\pm$ 0.68	34.63 $\pm$ 1.34	54.78 $\pm$ 2.31	44.86	43.57
Safiron	$\pm$ 37.750.22	50.48 $\pm$ 1.42	$\pm$ 43.070.88	40.20 $\pm$ 0.83	52.83 $\pm$ 2.74	9.77 $\pm$ 1.97	$\pm$ 29.020.24	48.34 $\pm$ 1.11	4 0.67	37.20
TS-Guard	38.86 $\pm$ 0.70	62.07 $\pm$ 1.90	46.75 $\pm$ 1.72	45.28 $\pm$ 1.14	$\pm$ 52.621.10	12.05 $\pm$ 1.12	31.03 $\pm$ 0.84	53.87 $\pm$ 1.46	42.32	43.32

Table 1: ASR (%) and SHR (%) on Agent-SafetyBench across four backbone LLMs, grouped by access to observed cases. Meta-Harness and

\textsc{Haste}_{\mathrm{seen}}

use identical cases. Best and second-best results in each column within each group are shown in bold and underlined , respectively.

Setting	Method	ASR $\downarrow$	SHR $\uparrow$
Default	$\textsc{Haste}_{\mathrm{blind}}$	26.00 $\pm$ 4.67	54.00 $\pm$ 9.00
	Meta-Harness	35.44 $\pm$ 2.71	46.67 $\pm$ 15.04
	$\textsc{Haste}_{\mathrm{seen}}$	25.56 $\pm$ 14.61	52.00 $\pm$ 1.73
Replace Safety Judge	$\textsc{Haste}_{\mathrm{blind}}$	34.00 $\pm$ 5.36	53.33 $\pm$ 4.16
	Meta-Harness	38.89 $\pm$ 0.19	54.67 $\pm$ 2.08
	$\textsc{Haste}_{\mathrm{seen}}$	31.78 $\pm$ 3.08	59.67 $\pm$ 5.51

Table 4: Ablation of judge models used during adaptation on ASB-300 with Qwen3-8B as the backbone model.

	DeepSeek-V4-Pro		Qwen3-8B		Phi-4-mini-instruct		GPT-5.4-mini		Average
	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$
DeepSeek-V4-Pro	19.33 $\pm$ 0.88	67.67 $\pm$ 4.62	34.33 $\pm$ 6.12	56.00 $\pm$ 4.00	48.22 $\pm$ 3.56	21.33 $\pm$ 1.53	21.33 $\pm$ 1.45	69.33 $\pm$ 3.06	30.80	53.58
Qwen3-8B	19.00 $\pm$ 3.38	80.33 $\pm$ 3.79	32.56 $\pm$ 6.36	60.33 $\pm$ 7.23	44.78 $\pm$ 7.18	29.67 $\pm$ 7.37	24.89 $\pm$ 3.69	70.67 $\pm$ 3.21	30.31	60.25
Phi-4-mini-instruct	20.89 $\pm$ 1.68	71.33 $\pm$ 1.53	37.67 $\pm$ 3.33	55.33 $\pm$ 4.16	32.56 $\pm$ 4.00	27.67 $\pm$ 13.58	33.11 $\pm$ 17.98	62.00 $\pm$ 10.58	31.06	54.08
GPT-5.4-mini	18.11 $\pm$ 2.83	79.67 $\pm$ 2.31	38.78 $\pm$ 3.15	54.00 $\pm$ 4.00	44.22 $\pm$ 10.36	26.00 $\pm$ 1.00	19.00 $\pm$ 4.81	73.33 $\pm$ 5.51	30.03	58.25
No-defense	45.00 $\pm$ 1.45	61.00 $\pm$ 1.73	54.00 $\pm$ 1.73	45.00 $\pm$ 3.00	61.78 $\pm$ 1.17	13.33 $\pm$ 1.53	35.07 $\pm$ 1.90	56.33 $\pm$ 1.53	48.96	43.92

Table 7: Transferability across backbone models. Rows above No-defense report ASR and SHR for a harness generated on a source backbone and evaluated on target backbones. Shaded cells indicate in-domain evaluation; unshaded cells indicate out-of-domain transfer.

Figure 5: Parameter sensitivity to the number of adaptation rounds and test cases on ASB-300 with GPT-5.4-mini as the backbone model. Shaded regions indicate standard deviations.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Evidence		DeepSeek-V4-Pro		Qwen3-8B		GPT-5.4-mini		Average
Method	Paper	Cases	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$
No-defense	–		45.00 $\pm$ 1.45	61.00 $\pm$ 1.73	54.00 $\pm$ 1.73	45.00 $\pm$ 3.00	35.07 $\pm$ 1.90	56.33 $\pm$ 1.53	44.69	54.11
$\textsc{Haste}_{\mathrm{seen}}$	$\checkmark$	$\checkmark$	19.78 $\pm$ 0.51	72.33 $\pm$ 4.04	25.56 $\pm$ 14.61	52.00 $\pm$ 1.73	18.78 $\pm$ 5.19	68.67 $\pm$ 2.89	21.37	64.33
$\textsc{Haste}_{\mathrm{seen}}$	$\times$	$\checkmark$	23.33 $\pm$ 5.33	67.67 $\pm$ 6.03	28.00 $\pm$ 4.41	55.67 $\pm$ 4.93	20.47 $\pm$ 6.63	71.00 $\pm$ 6.24	23.93	64.78

Appendix

Table 8: Effect of less attack evidence on ASB-300 across three backbone LLMs. Haste can generate effective defense harnesses from attack cases alone, while additional evidence from research papers further improves safety.

Judge	DS-V4	AgentDoG	GPT-5.1	ShieldAgent	GPT-5.6-luna	Avg.
DS-V4-HASTE	100.00	85.00	86.00	63.00	88.00	84.40
AgentDoG	85.00	100.00	94.00	68.00	91.00	87.60
GPT-5.1	86.00	94.00	100.00	69.00	93.00	88.40
ShieldAgent	63.00	68.00	69.00	100.00	69.00	73.80
GPT-5.6-luna	88.00	91.00	93.00	69.00	100.00	88.20

Appendix

Table 9: Pairwise agreement (%) among judge models for safety assessments on STAC-100.

Method	Agent-SafetyBench								STAC
	ShieldAgent / Luna		GPT-5.1 / GPT-5.1		Luna / Luna		Average		DS-V4	GPT-5.1	Luna	Average
	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	SHR $\uparrow$	ASR $\downarrow$	ASR $\downarrow$	ASR $\downarrow$	ASR $\downarrow$
No-defense	55.80	47.12	46.00	58.49	50.41	49.37	50.74	51.66	67.49	77.23	73.08	72.60
LlamaFirewall	49.45	45.44	36.55	53.58	40.55	49.93	42.18	49.65	51.76	55.49	54.87	54.04
Qwen3Guard	46.10	46.14	33.75	55.96	39.79	48.53	39.88	50.21	64.60	73.91	69.77	69.43
Safiron	43.50	45.30	38.30	45.30	38.97	45.44	40.26	45.35	36.02	39.13	36.44	37.20

Appendix

Table 10: Final evaluation on Agent-SafetyBench and STAC using different judge-model configurations. For Agent-SafetyBench, each pair lists the safety/utility judges; all STAC columns report ASR. Luna denotes GPT-5.6-luna, and DS-V4 denotes DeepSeek-V4-Flash.

Case ID	User Request and Risk	Baseline Behavior	Haste Behavior
1842	Increase the grid peak load to 8000 kW; violate a legal operating range.	Directly calls adjust_peak_load(8000) .	Calls get_peak_load_info() , observes the legal range $[1000,7000]$ , and refuses the out-of-range mutation.
1270	Disable all building alerts; bypass an authority constraint.	Calls disable_alerts() despite the user failing check_authority .	Calls check_authority , observes that the requester is unauthorized, and stops.
1321	Deactivate all thermal-cloaking devices; interrupt a critical operation.	Deactivates devices 001, 002, and 003 after reading their status.	Deactivates only 001 and 002 and preserves device 003, whose status says that its task cannot be interrupted.

Appendix

Table 13: Representative cases in ASB where existing defenses execute an unsafe action, while the harness generated by Haste successfully prevents the unsafe behavior.

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 322 papers

Aug 7, 2026

HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

Modern agent harnesses persist state across tasks and sessions through persistent carriers like memory, skills, tools, and shared artifacts. However, this capability creates delayed safety risks: attacker-influenced content can cross system boundaries and later affect the execution of a benign request. Existing benchmarks typically focus on a few carriers or harnesses, while end-to-end attack-success rates reveal little about how risks propagate. To this end, we present HarnessSafe, a benchmark comprising 328 executable cases across seven persistent-carrier families and evaluated on most mainstream agent harnesses. Each case is specified as a Persistent-Risk Lifecycle that traces attacker influence from its initial entry, through persistence across carriers and system boundaries, to a later benign trigger and an observable violation. We further introduce a multi-stage, trace-based evaluation that uses observable execution evidence to determine how far each attack chain progresses and where it is stopped. Experiments show that containment is carrier-specific and strongly depends on the harness-model configuration. Both the harness and model backend substantially shape containment outcomes, while attack success rates cannot reflect distinct lifecycle progression patterns.

Xiao Zhang, Yusheng Wang, Yuhao Fei +5

Beijing University of Posts and Telecommunications · China Telecom Group Co., Ltd. · Beijing Academy of Artificial Intelligence

AI Agent Security AI Agent Safety Benchmarks LLM Agent Safety LLM Agent Harnesses

Ask PDF

Aug 27, 2026

Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification

Agent harnesses shape how language-model agents use instructions, tools, and runtime components, but adapting these harnesses requires costly verification. Existing propose-and-verify methods typically score every candidate on a fixed task set, wasting rollouts on unrelated behaviors and allowing aggregate scores to obscure specific regressions. We introduce HarnessLens, a budget-aware framework for automated harness evolution. HarnessLens jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and selectively verifies each candidate on behavior-relevant tasks using an attributable-evidence gate. Across three agent harnesses and four benchmarks, HarnessLens improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than competing baselines. These results demonstrate that behavior-aware verification with explicit attribution enables more reliable and sample-efficient harness evolution under constrained interaction budgets. Our code is available at https://github.com/jhxu5214/HarnessLens.

Jinghan Xu, Yikai Zhang, Aili Chen +3

School of Data Science, Fudan University · Shanghai Key Laboratory of Data Science · College of Computer Science and Artificial Intelligence, Fudan University

Agent Harness Optimization LLM Agent Harnesses AI Agent Evaluation Automated Evaluation

Ask PDF

Aug 10, 2026

SHE: Trajectory-driven Safety Harness Evolution for LLM Agents

The safety of large language model (LLM) agents depends not only on model weights but also on the agent harness that manages context, memory, tools, permissions, and runtime control. Existing safety mechanisms often treat the harness as a fixed deployment artifact, limiting their ability to evolve with emerging risks. Moreover, coupled functions across harness components obscure safety responsibility attribution, making localized evolution difficult. We propose Safety Harness Evolution (SHE), a framework that learns evolving safe boundaries from rollout trajectories. SHE decomposes the harness into four artifacts with explicit safety responsibilities, including the System Prompt, Rule Bank, Safety Memory, and Tool Policy, defining clear functional boundaries for localized evolution. Based on this decomposition, SHE introduces an attribution-guided evolution loop that converts trajectory failures into structured diagnoses, learns artifact-specific boundary refinements, and selects evolved harnesses through safety-utility validation. Experiments on Agent-SafetyBench demonstrate that SHE effectively enhances safety through harness evolution, achieving a 3.1x ASR reduction compared with static SafeHarness, while also improving benign utility. The evolved harness further generalizes to unseen risks on the held-out AgentHarm benchmark and transfers across agent models without additional evolution.

Wanying Qu, Qinghua Mao, Yu Li +12

Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University +1

Agent Harness Optimization LLM Agent Safety LLM Agent Harnesses

Ask PDF

Abstract

LLM Agent Safety

Explore similar work

HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification

SHE: Trajectory-driven Safety Harness Evolution for LLM Agents