HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention

Han Luo; Bingbing Wen; Guang Yang; Zora Zhiruo Wang; Pan Lu; Lucy Lu Wang

cs.AIOct 5, 2026

HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention

Authors: Han Luo, Bingbing Wen, Guang Yang, Zora Zhiruo Wang, Pan Lu, Lucy Lu Wang

Organizations: University of Washington · University of Leeds · Carnegie Mellon University · Stanford University · Allen Institute for AI

Abstract

Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists. Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.

Figures & tables

Figure 1: Harness–environment co-evolution. Given an environment pool of feasible and infeasible tasks and current harness, paired rollouts and failure diagnosis are used to iteratively evolve both the environment pool and harness. Environment evolution (top) constructs, validates, and selects new feasible–infeasible task pairs to challenge the current agent and expand the training pool. Harness evolution (bottom) similarly refines the harness iteratively on the expanded task pool, with each updated harness selected from candidates using performance on a validation set.

Static harness baselines
Method	Abstain $\uparrow$	Act $\uparrow$	Pair $\uparrow$
$H_{0}$	61.7	68.3	48.3
$H_{0}$ + abstention prompt	68.3	63.3	46.7
Harness optimization baselines
Harness-only evolution	76.7	71.7	60.0
Task-augmented harness evolution	80.0	75.0	65.0

Table 1: Comparison with baseline harness optimization strategies on the fixed evaluation set (%). All methods use the same base model and evaluation tasks.

Method	$\mathbf{T_{0}}$	$\mathbf{T_{1}}$	$\mathbf{T_{2}}$	$\mathbf{T_{3}}$	$\mathbf{T_{4}}$	$\mathbf{T_{5}}$	$\mathbf{T_{6}}$
Harness-only evolution	90.0 / 80.0	84.0 / 76.0	76.7 / 70.0	77.1 / 71.4	70.0 / 67.5	66.7 / 66.7	64.0 / 66.0
Task-augmented harness evolution	90.0 / 85.0	88.0 / 84.0	80.0 / 80.0	80.0 / 80.0	75.0 / 77.5	75.6 / 77.8	74.0 / 76.0
Task generation from $H_{0}$ failures	95.0 / 90.0	96.0 / 92.0	86.7 / 86.7	85.7 / 88.6	82.5 / 85.0	77.8 / 84.4	76.0 / 82.0
Co-evolution (ours)	95.0 / 90.0	96.0 / 92.0	90.0 / 90.0	91.4 / 91.4	90.0 / 90.0	86.7 / 88.9	86.0 / 88.0

Table 2: Performance across cumulative training pools. Each cell reports Abstain / Act (%).

T_{0}

denotes the initial training pool, and

T_{1}

–

T_{6}

denote cumulative pools after successive task expansions. The same final harness for each method is evaluated across all pools.

Figure 2: Performance on fixed task batches during co-evolution. From left to right, the panels show Abstain, Pair, and Act. Each curve tracks a fixed batch of tasks across harness updates.

B_{0}

is the initial training batch, and

B_{i}

(

i\geq 1

) contains the new tasks introduced in round

i

. These batches do not overlap, and

T_{i}=\bigcup_{j=0}^{i}B_{j}

is the cumulative training pool.

H_{i}

denotes the harness selected at the end of round

i

Figure 3: Abstention and paired success rates versus API cost on hera-bench . Colors identify models; arrows connect H 0 (circles) to the fixed Final Harness (diamonds).

Figure 4: Example of harness evolution across successive updates. Starting from the base executor

H_{0}

, each subsequent harness introduces or refines logic in response to diagnosed failures.

H_{1}

adds an evidence-based decision gate to distinguish recoverable uncertainty from conditions that warrant abstention;

H_{2}

introduces cross-source evidence reconciliation;

H_{3}

jointly verifies interacting task constraints before consequential actions; and

H_{4}

extends this verification to authorization relationships and cumulative privacy constraints. Green regions indicate newly introduced or refined logic relative to the preceding harness.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: End-to-End Environment–Task Construction Pipeline. The pipeline consists of three stages: environment construction, task generation, and environment mutation.

Model	Requested API identifier	Serving route	Reasoning
GPT-5.6-Luna	gpt-5.6-luna	Responses API	Default
GPT-5.6-Terra	openai/gpt-5.6-terra	openai	High
GPT-5.6-Sol	openai/gpt-5.6-sol	openai	High
GPT-6-Astra	openai/gpt-6-astra	openai	High
Claude Opus 5	anthropic/claude-opus-5	anthropic	High
Claude Sonnet 5	anthropic/claude-sonnet-5	anthropic	High

Appendix

Table 4: Models and inference configurations. “Default” indicates that no reasoning parameter is explicitly supplied. OpenRouter provider identifiers are shown for routed models.

Method	$\mathbf{T_{0}}$	$\mathbf{T_{1}}$	$\mathbf{T_{2}}$	$\mathbf{T_{3}}$	$\mathbf{T_{4}}$	$\mathbf{T_{5}}$	$\mathbf{T_{6}}$
Harness-only evolution	75.0	68.0	60.0	60.0	52.5	48.9	46.0
Task-augmented harness evolution	75.0	72.0	66.7	65.7	60.0	60.0	58.0
Task generation from $H_{0}$ failures	85.0	88.0	80.0	80.0	75.0	71.1	68.0
Co-evolution (ours)	85.0	88.0	83.3	85.7	82.5	80.0	78.0

Appendix

Table 6: Paired success (Pair, %) using the same final harnesses and cumulative training pools as in Table 2 .

Figure 6: Primary changes across rounds of environment evolution. Each bar summarizes the five task pairs retained in that round.

Panel of Figure 7 — Figure 7: Weather task and reference trajectory. The reference trajectory is obtained by executing a reference policy in the weather environment. The policy first looks up the Los Angeles weather point, then uses the returned county and forecast identifiers to retrieve the relevant stations and forecast periods.

Panel of Figure 8 — Figure 8: Weather task and reference trajectory in the infeasible instance. The task instruction and public tools remain unchanged. The forecast response marks the Saturday daytime precipitation probability as unavailable. The reference trajectory still establishes the closest station and the available forecast evidence, but cannot determine whether the precipitation threshold is exceeded. Resource identifiers and observations are abbreviated for presentation.

Explore similar work

AgentAbstain: Do LLM Agents Know When Not to Act?
Jul 11, 2026Xun Liu, Yi Evie Zhang, Vira Kasprova +5Large Language Model Agents Language-Model Agents
University of Illinois Urbana-Champaign
Agentic Abstention: Do Agents Know When to Stop Instead of Act?
Jun 27, 2026Han Luo, Bingbing Wen, Lucy Lu WangLanguage-Model Agents Abstention
University of Leeds · Southwest Jiaotong University · University of Washington +1
Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents
Sep 22, 2026Laizhen Li, Jiarui Li, Juanjuan Zhao +4Agent Harness Scaffolds
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Shenzhen University of Advanced Technology +1

Jul 11, 2026, cs.AIJ/K move · Enter open · S save

AgentAbstain: Do LLM Agents Know When Not to Act?

Xun Liu, Yi Evie Zhang, Vira Kasprova, Parisa Rabbani+4

University of Illinois Urbana-Champaign

Large Language Model Agents Language-Model Agents Abstention Taxonomy

Open paper AskPDF arXiv

Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain. This gap poses real risks: under ambiguity, conflicting constraints, or tool failures, agents may execute unintended and irreversible actions. To close this gap, we present the first systematic evaluation framework for agentic abstention: the calibrated ability of tool-using LLM agents to recognize when not to act. At its core, AgentAbstain is a paired-task benchmark built on an agent-native taxonomy of 8 abstention scenarios across pre-execution reasoning and runtime discovery. It contains 263 paired tasks across 42 executable sandbox environments, where each pair consists of a should-act task and a should-abstain variant produced through a controlled perturbation to the instruction, tool, or environment state. To scale this paired design and resist data contamination, we propose AbstainGen, a fully automated pipeline that synthesizes sandbox environments and generates paired tasks end-to-end, validated by deterministic replay and semantic LLM judges; fresh task instances can be regenerated on demand, and three independent annotators rate 94-98% of sampled tasks as well-designed. Across 17 frontier LLMs in 4 agent harnesses, the best agent (Gemini 3.1 Pro) achieves only 59.5% paired accuracy (correct on both the act and abstain sides of each paired task). More importantly, abstention capability is largely independent of general task-solving capability, indicating that scaling task-solving alone will not close this gap. We further identify failure modes such as post-hoc abstention, in which agents execute irreversible actions before recognizing abstention triggers. Our code and dataset are open-sourced at agentabstain.github.io.

Abstract

Explore similar work

AgentAbstain: Do LLM Agents Know When Not to Act?

Agentic Abstention: Do Agents Know When to Stop Instead of Act?

Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents