Self-Designed Evaluators and Warm Memory for Long-Horizon Agents

Saeid Asgari; Emre Kiciman; Leonardo de Oliveira Nunes; Ranveer Chandra

cs.AISep 27, 2026

Self-Designed Evaluators and Warm Memory for Long-Horizon Agents

Authors: Saeid Asgari, Emre Kiciman, Leonardo de Oliveira Nunes, Ranveer Chandra

Organizations: Microsoft

Abstract

A tool-using language-model agent deployed over a long stream of tasks receives no reward, so it cannot tell whether it succeeded, cannot safely retry, and cannot label the experience it needs to improve. We present SelfSuite, in which the agent's own base model, given only the world's public materials, designs a small evaluation suite of weighted judges and grounded per-task briefs, freezes it, and uses it to gate a keep-best retry and to label a typed, outcome-tracked memory. On matched five-repeat benchmarks over tau2-bench and AppWorld, SelfSuite scores above the plain agent without any labels, matches methods given ten expert labels on tau2-bench, and trails Agentic Context Engineering (ACE) on AppWorld, where code execution gives a direct success signal. In an ablation campaign run on the same tasks, it is above label-free ACE in every repeat, and the gated second attempt is the only component whose removal hurts in every repeat. We also simulate a subject-matter expert who grades ten onboarding tasks per world. Using those labels to calibrate SelfSuite's evaluator gives a small, consistent gain, and using them to warm up ACE's memory lifts ACE to tie calibrated SelfSuite. A single-run study on a second model family shows the same ordering.

Figures & tables

Comparison	Group	$\Delta$ (pp)	95% CI, $p$
SelfSuite + calibrated $-$ plain agent	overall	$+8.0\pm 4.7$	$[+2.2,\,+13.8]$ , $0.019$
	tau2	$+4.8\pm 4.4$	$[-0.6,\,+10.2]$ , $0.070$
	AppWorld	$+11.2\pm 7.7$	$[+1.6,\,+20.8]$ , $0.031$
SelfSuite , fresh $-$ plain agent	overall	$+6.0\pm 5.1$	$[-0.3,\,+12.3]$ , $0.058$
ACE + warm playbook $-$ plain agent	overall	$+7.6\pm 4.3$	$[+2.2,\,+13.0]$ , $0.017$
ACE, cold $-$ plain agent	overall	$+4.8\pm 8.6$	$[-5.8,\,+15.4]$ , $0.278$

Table 1: Paired per-repeat differences (campaign A). Mean

\pm

SD of the five per-repeat differences in percentage points, 95%

t

CI, and paired

t

-test

p

-value.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

World	$S_{0}$ gate agr.	Proposal gate agr.	Decision	Frozen agg., $\tau$
tau2 airline	100.0%	75.0%	rejected, $S_{0}$ kept	min, 0.99
tau2 retail	66.7%	83.3%	accepted	min, 0.99
tau2 telecom	100.0%	100.0%	no improvement possible, $S_{0}$ kept	weighted, 0.85
AppWorld normal	100.0%	100.0%	no improvement possible, $S_{0}$ kept	min, 0.99
AppWorld challenge	100.0%	100.0%	no improvement possible, $S_{0}$ kept	weighted, 0.85

Appendix

Table 2: Warm-up-v2 calibration outcome per world (campaign A). Agreement of the authored suite (

S_{0}

) and of the calibrated proposal with the four gate labels, and the frozen result.

Arm	Signal	Tau2	AppWorld	Overall
Plain agent	none	$74.4\pm 5.4$	$78.4\pm 9.6$	$76.4\pm 6.2$
ACE re-impl. collapsing, cold	self-graded	$72.8\pm 10.7$	$90.4\pm 3.6$	$81.6\pm 4.6$
ACE re-impl. collapsing, cold	calibrated evaluator	$72.8\pm 9.6$	$92.0\pm 4.0$	$82.4\pm 3.0$
ACE re-impl. collapsing, cold	GT oracle	$72.0\pm 4.9$	$88.8\pm 5.2$	$80.4\pm 3.6$
ACE re-impl. collapsing, warm, frozen	none	$72.8\pm 7.2$	$88.8\pm 5.2$	$80.8\pm 4.6$
ACE re-impl. collapsing, warm	self-graded	$69.6\pm 8.3$	$88.0\pm 7.5$	$78.8\pm 6.1$

Appendix

Table 3: All arms of campaign A (GPT-5.5, five matched repeats). “Oracle” rows read the true label on every task and are not deployable. “Collapsing” and “delta” are our re-implementation of ACE, with and without a full playbook rewrite after every task; rows labelled plain “ACE” use the memory machinery released by the ACE authors. Warm playbooks are fitted from the ten warm-up labels.

World	Judges, agg., $\tau$	Judged pass	GT	Retried	FP	FN	Discr.	Retry gain	Oracle
Airline (40)	2, min, 0.99	67.5%	64.0	37.0%	39%	26%	$+0.18$	$+1.5$	66.0
Retail (104)	1, min, 0.99	86.5%	84.2	26.9%	53%	25%	$+0.13$	$+1.0$	88.5
Telecom (104)	3, weighted, 0.85	72.3%	79.6	38.8%	13%	23%	$+0.38$	$+4.6$	84.2
AppWorld normal (158)	2, min, 0.99	84.7%	90.8	20.0%	29%	23%	$+0.43$	$+1.5$	91.1

Appendix

Table 4: Operating point of the calibrated suites (campaign D; mean over five deployments on the whole task sets), values plotted in Figure 4 . Judges and aggregation as frozen after calibration; per-attempt false-positive (FP) and false-negative (FN) rates of the suite verdict against ground truth; discrimination = mean judge score on GT-pass minus GT-fail attempts; retry gain = deployed minus first-attempt success; oracle = success if the better of the two attempts had always been selected.

Model	World	Plain agent	ACE, no GT	SelfSuite
GPT-5.5 @low	AppWorld normal (168)	87.5	90.5	91.7
	AppWorld challenge (417)	64.5	71.2	77.7
	tau2 airline (50)	64.0	70.0	66.0
	tau2 retail (114)	83.3	83.3	83.3
	tau2 telecom (114)	70.2	69.3	75.4
Opus-4.8 @off	AppWorld normal (168)	70.8	86.9	92.3

Appendix

Table 5: Two-model single-run study (campaign C), values plotted in Figure 5 . Success (%) on the full suites; one run per cell. Bold = best in row.

Explore similar work

LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
May 12, 2026Di Wu, Zixiang Ji, Asmi Kawatkar +4Long-Term Agent Memory Long-Term Memory
University of California, Los Angeles
EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer
Jul 6, 2026Xingze Gao, Chuanrui Hu, Hongda Chen +9Self-Evolving Agents Agentic Benchmarks
1Anhui University · 2EverMind, Shanda Group · 3Southeast University
Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
Aug 13, 2026Yiwei Li, Wanli Yang, Hexiang Tan +10Long-Horizon Agents Autonomous Agents
1Meituan · University of Chinese Academy of Sciences

May 12, 2026, cs.CLJ/K move · Enter open · S save

LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan+3

University of California, Los Angeles

Long-Term Agent Memory Long-Term Memory Memoryagentbench Single-Agent Baselines Memes

Open paper AskPDF arXiv

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success, leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. To address this gap, we introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments. LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. Questions are paired with history trajectories containing up to 500 trajectories and 115M tokens. We use a context gathering formulation: memory systems consume history trajectories and return compact evidence for downstream question answering. We propose a suite of two memory methods: AgentRunbook-R, an efficient RAG-based memory with knowledge pools for raw state observations, events, and strategy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. Experiments show that AgentRunbook-C achieves the best performance with 72.5% average accuracy, outperforming the strongest RAG baseline (48.5%) and the off-the-shelf coding agent baseline (69.3%). Despite the strong performance gains, coding agent based methods have high latency costs. While AgentRunbook-C advances the accuracy-latency Pareto frontier, substantial room for improvement remains. Together, these results establish LME-V2 as a challenging testbed for developing long-term memory systems for environment experience.

Abstract

Explore similar work

LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development