Co-Evolving Agents: Learning from Failures as Hard Negatives

Yeonsung Jung; Trilok Padhi; Sina Shaham; Dipika Khullar; Joonhyun Jeong; Ninareh Mehrabi; Eunho Yang

cs.AINov 27, 2025

Co-Evolving Agents: Learning from Failures as Hard Negatives

Authors: Yeonsung Jung, Trilok Padhi, Sina Shaham, Dipika Khullar, Joonhyun Jeong, Ninareh Mehrabi, Eunho Yang

Organizations: KAIST · Georgia State University · Independent Researcher · NAVER Cloud · AITRICS

Abstract

Self-evolving agents improve their performance on long-horizon tasks by learning from their own interactions with an environment. A common approach uses the resulting failed trajectories as negatives for preference training. However, collecting an agent's own failures does not ensure that they provide informative supervision for further improvement. Obvious failures may be easy to reject without learning to identify errors in more plausible attempts. These plausible but incorrect trajectories can serve as hard negatives, helping agents learn to distinguish successful behavior from failures with less obvious errors. To enable agents to generate and learn from such informative negatives, we propose a co-evolving framework that couples a target agent's self-improvement with a failure agent trained to generate hard negatives. The failure agent learns exclusively from both agents' failed trajectories to generate higher-reward failures, which the target agent uses as negatives for preference training. As the target policy improves, its failures provide new training data for the failure agent. The updated failure agent, in turn, provides new negatives for target training, allowing negative generation to adapt alongside the target policy. Across online shopping, scientific reasoning, and interactive SQL querying, our framework improves average task reward by 5.7 over the baseline. These results demonstrate the value of learning to generate and use informative failures for agent self-improvement. We will make our code publicly available.

Figures & tables

Figure 1: Overview of our co-evolving framework. Both agents generate trajectories for repeated training. The target agent prefers expert demonstrations over failures; the failure agent prefers higher-reward failures over lower-reward ones, regardless of source. The highlighted hard negative is rejected during target agent training but preferred during failure agent training. Colors denote trajectory sources, bars show rewards, and dashed loops indicate repeated updates.

Dataset	Train	Test		Action Space	Max Turns
Dataset	Train	Seen	Unseen	Action Space	Max Turns
WebShop	1,824	200	–	8	10
ScienceWorld	1,483	194	211	16	100
InterCodeSQL	1,500	200	–	$\infty$	10

Table 1: Benchmark statistics. ScienceWorld includes seen and unseen evaluation splits. Max Turns denotes the interaction limit per episode.

Model	Method	WebShop	ScienceWorld		InterCodeSQL	Avg.
Model	Method	WebShop	Seen	Unseen	InterCodeSQL	Avg.
Qwen3-4B	SFT	66.00	83.30	67.90	58.10	68.83
	ETO	72.90	80.70	83.10	60.70	74.35
	Ours	74.10	87.60	88.30	63.00	78.25
Qwen3.5-9B	SFT	55.15	79.24	73.80	68.59	69.20
	Agent-R	55.24	69.68	56.66	65.31	61.72

Table 2: Main results on WebShop, ScienceWorld, and InterCodeSQL across three base models. Results are average task rewards on a

0

–

100

scale.

Model	Method	WebShop	ScienceWorld		InterCodeSQL	Avg.
Model	Method	WebShop	Seen	Unseen	InterCodeSQL	Avg.
Qwen3.5-9B	ETO	68.06	80.47	75.40	68.94	73.22
	SRLM	54.09	78.11	74.09	66.34	68.16
	Best-of- $N$	71.14	82.87	75.94	65.02	73.74
	Ours	71.30	88.60	84.72	70.63	78.81
Gemma-4-12B-it	ETO	71.67	77.11	69.59	70.36	72.18

Table 3: Comparison with methods that use multiple rollouts during training. SRLM and Best-of-

N

use

N=4

; ETO and Ours use one rollout per instruction per generating agent.

Figure 2: Failure rewards and early target performance. (a) Failure counts by reward bin for Qwen3.5-9B (top) and Gemma-4-12B-it (bottom). Each panel uses its own count scale. (b) Target performance gains over ETO at iteration 1. Reward bins are used only for analysis.

Model	Method	WebShop	ScienceWorld		InterCodeSQL	Avg.
Model	Method	WebShop	Seen	Unseen	InterCodeSQL	Avg.
Qwen3.5-9B	SFT	55.15	79.24	73.80	68.59	69.20
	ETO	68.06	80.47	75.40	68.94	73.22
	Ours	71.30	88.60	84.72	70.63	78.81
	w/o Co-Evolution	71.10	87.05	79.52	70.27	76.99
	w/o Failure Agent	70.93	68.62	63.03	68.71	67.82

Table 4: Ablations of the failure agent and co-evolution. w/o Failure Agent replaces the failure agent with a positive agent trained with the target objective; w/o Co-Evolution freezes the failure agent after iteration 1.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Adaptation	Models	WebShop	ScienceWorld		InterCodeSQL	Avg.
Adaptation	Models	WebShop	Seen	Unseen	InterCodeSQL	Avg.
In-context	GPT-4	63.2	42.9	38.1	38.5	45.7
In-context	GPT-3.5-Turbo	62.4	7.9	10.5	37.8	29.7
Fine-tuning	Llama-2-7B-Chat + SFT	58.7	53.4	49.9	26.5	47.1
	Llama-2-7B-Chat + PPO	64.2	59.4	51.7	32.4	51.9
	Llama-2-7B-Chat + RFT	61.3	71.6	54.3	25.6	53.2

Appendix

Table 6: Additional results for Llama-2-7B-Chat. GPT baselines use in-context learning. Avg. is the mean over the four evaluation settings.

Models	WebShop	ScienceWorld	Avg.
Llama-2-13B-chat + SFT	61.8	64.5	63.2
Llama-2-13B-chat + AgentLM ( Zeng et al., 2024 )	39.5	10.7	25.1
Llama-2-13B-chat + Agent-FLAN ( Chen et al., 2024 )	40.4	28.6	34.5
Llama-2-13B-chat + ETO	66.0	72.6	69.3
Llama-2-13B-chat + Ours	69.2	74.5	71.9

Appendix

Table 7: Additional results for Llama-2-13B-Chat on WebShop and ScienceWorld. Avg. is the mean over the two environments.

Explore similar work

HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution
Sep 1, 2026Wen Jiang, Mingmin Chu, Yimeng Tian +6Self-Evolving Agents Self-Evolution
ICT AI Competence Center, Huawei Technologies, Shanghai, China
Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents
Jun 30, 2026Xueqiao Sun, Xiaohan Wang, Ludwig Schmidt +2Self-Improving Agents Recursive Self-Improvement
1Stanford University, Stanford, CA 94305, USA
Self-Improvements in Modern Agentic Systems: A Survey
Jul 14, 2026Zhe Ren, Yimeng Chen, Dandan Guo +9Self-Improving Agents Recursive Self-Improvement
School of Artificial Intelligence, Jilin University · King Abdullah University of Science and Technology (KAUST) · Independent Researcher +2

Sep 1, 2026, cs.LGJ/K move · Enter open · S save

HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution

Wen Jiang, Mingmin Chu, Yimeng Tian, Qianxin Zhang+5

ICT AI Competence Center, Huawei Technologies, Shanghai, China

Self-Evolving Agents Self-Evolution Evolutionary Optimization Methods Gating Shortcut Learning

Open paper AskPDF arXiv

Self-evolving agents advance toward autonomy by optimizing their harness---prompts, skills, tools, and execution logic---based on environmental feedback. This paradigm, however, is hampered by three challenges: \textit{credit assignment failure}, where terminal success/failure feedback makes it ambiguous which step caused the error; \textit{shortcut learning}, where agents memorize task-specific patterns rather than acquire generalizable capabilities; and \textit{catastrophic forgetting}, where unguarded updates degrade previously acquired competence. In this paper, we introduce HarnessEvolve, a self-evolving framework that learns from reference trajectories to achieve reliable agent self-evolution. HarnessEvolve decouples the execution agent from the evolutionary pipeline, assigning execution, evaluation, optimization, and gating to independent agent modules, enabling generalizable and stable harness improvements. Specifically, HarnessEvolve overcomes credit assignment failure by generating reference trajectories (execution paths produced when given the ground-truth answers) and aligning failed executions against them to extract error signals, which are clustered to reveal systematic failure patterns. To prevent shortcut learning and catastrophic forgetting, candidate harness updates must pass two gates: a quality gate that filters data leakage and prompt bloat, and a performance gate that accepts each update if it improves on the current batch without degrading recent batches, with epoch-end validation on a held-out set selecting the best-performing accepted agent snapshot. We conduct extensive experiments on several benchmarks spanning open-domain and enterprise scenarios, using different models and agent frameworks. Results demonstrate that HarnessEvolve consistently outperforms state-of-the-art baselines across all benchmarks and settings, confirming reliability across task domains.

Abstract

Explore similar work

HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution

Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents

Self-Improvements in Modern Agentic Systems: A Survey